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Preface 



T he central idea of evidence-based education — that education policy 
and practice ought to be fashioned based on what is known from 
rigorous research — offers a compelling way to approach reform 
efforts. Recent federal trends reflect a growing enthusiasm for such change. 
Most visibly, the No Child Left Behind Act of 200 1 requires that “scientifi- 
cally based [education] research” drive the use of federal education funds at 
the state and local levels. This emphasis is also reflected in a number of 
government and nongovernment initiatives across the country. As consen- 
sus builds around the goals of evidence-based education, consideration of 
what it will take to make it a reality becomes the crucial next step. 

In this context, the Center for Education of the National Research 
Council (NRC) has undertaken a series of activities to address issues related 
to the quality of scientific education research. 1 In 2002, the NRC released 
Scientific Research in Education (National Research Council, 2002), a re- 
port designed to articulate the nature of scientific education research and to 
guide efforts aimed at improving its quality. Building on this work, the 
Committee on Research in Education was convened to advance an im- 
proved understanding of a scientific approach to addressing education prob- 



1 Other NRC efforts — especially the line of work that culminated in the recent report 
Strategic Education Research Partnership (National Research Council, 2003) — offer insights 
and advice about ways to advance research utilization more broadly. 
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viii PREFACE 

lems; to engage the field of education research in action-oriented dialogue 
about how to further the accumulation of scientific knowledge; and to 
coordinate, support, and promote cross-fertilization among NRC efforts in 
education research. 

The main locus of activity undertaken to meet these objectives was a 
year-long series of workshops to engage a range of education stakeholders 
in discussions about five key topics: 

• Peer Review in Federal Education Research Programs. This workshop 
focused on the purposes and practices of peer review in the federal agencies 
that fund education research. Federal officials and researchers considered a 
range of models used across the federal government to involve peers in the 
review of proposals for funding and discussed ways to foster high-quality 
scientific research. 

• Understanding and Promoting Knowledge Accumidation in Education: 
Tools and Strategies for Education Research. With a focus on how to build a 
coherent knowledge base in education research, researchers and federal of- 
ficials considered several elements of the research infrastructure, including 
tools, practices, models, and standards. Fundamental questions about what 
such a knowledge base might look like were also considered in this context. 

• Random Assignment Experimentation in Education: Implementation 
and Implications. The evidence-based education trend has brought to the 
fore decades of debate about the appropriateness of randomized field trials 
in education. Far less consideration has been devoted to the practical as- 
pects of conducting such studies in educational settings; this workshop 
featured detailed descriptions of studies using randomized field trials in 
education and reflections on how the current trend to fund more of these 
studies is influencing states, districts, and students. 

• Journal Practices in Publishing Education Research. Following the 
more general discussion of how to build a coherent knowledge base in 
education in a previous workshop, this event took up the specific case of 
journals that publish education research. Editors, publication committee 
members, and others involved in the production and use of journal articles 
considered ways to promote high-quality education research and to con- 
tribute to the larger body of knowledge about important areas of policy and 
practice. 

• Education Doctoral Programs for Future Leaders in Education Re- 
search. A final workshop focused on the professional development of edu- 
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PREFACE ix 

cation researchers, with a specific emphasis on doctoral programs in schools 
of education. Deans, graduate study coordinators, foundation officials, 
and policy makers came together to share observations and chart potential 
paths for progress. 

Additional information on each of these events can be found at http:// 
www7.nationalacademies.org/core/. 

This report is based on the first workshop in the series, on peer 
review in federal agencies that support education research, which took place 
on February 25-26, 2003, at the National Academies’ Keck Center in Wash- 
ington, DC. It summarizes common issues and ideas that emerged from 
the presentations and discussion during the workshop (see Appendix A for 
the workshop agenda and Appendix B for biographical sketches of the com- 
mittee members and speakers) and includes the committee’s conclusions 
and recommendations on how to strengthen peer review in federal agencies 
that support education research. 

This report would not have been possible without the help of the speak- 
ers who shared their expertise with the committee. We would like to thank 
each of them for their contributions: 

Diane August, August and Associates; Hilda Borko, University of Colo- 
rado, Boulder; Steven Breckler, National Science Foundation; Susan 
Chipman, Office of Naval Research; Domenic Cicchetti, Yale University; 
Louis Danielson, Office of Special Education Programs; Kenneth Dodge, 
Duke University; Edward Hackett, Arizona State University; Milton Hakel, 
Bowling Green State University; Teresa Levitin, National Institutes of 
Health; Penelope Peterson, Northwestern University; Edward Reddish, 
University of Maryland; Finbarr Sloane, National Science Foundation; 
Brent Stanfield, National Institutes of Health; Robert Sternberg, Yale Uni- 
versity; and Grover (Russ) Whitehurst, Institute of Education Sciences. 

Of course, without the generous support of our sponsors, neither the 
workshop nor this report would be possible. We extend our gratitude to 
the former National Educational Research Policy and Priorities Board and 
the Institute of Education Sciences, the William and Flora Hewlett Foun- 
dation, and the Spencer Foundation. 

We extend our thanks to each of the members of the Committee on 
Research in Education. We especially appreciate the efforts of the work- 
shop planning group, led by Jack Fletcher, who designed an outstanding 
event that has made a unique contribution to an important debate. Several 
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NRC staff played critical roles in shaping the workshop and deserve special 
recognition here: Meryl Bertenthal led the staff effort, substantively sup- 
ported by Tina Winters. R. Jason Rolsen provided the administrative and 
logistical support for the committee as well as for the event itself. And 
Patricia Morison offered general direction and guidance. Finally, we thank 
Chris McShane for her skillful editing of the manuscript. 

This report has been reviewed in draft form by individuals chosen for 
their diverse perspectives and technical expertise, in accordance with proce- 
dures approved by the NRC s Report Review Committee. The purpose of 
this independent review is to provide candid and critical comments that 
will assist the institution in making its published report as sound as pos- 
sible and to ensure that the report meets institutional standards for objec- 
tivity, evidence, and responsiveness to the study charge. The review com- 
ments and draft manuscript remain confidential to protect the integrity of 
the deliberative process. 

We wish to thank the following individuals for their review of this 
report: Michael Allen, Teaching Quality Policy Center, Education Com- 
mission of the States; Rolf Blank, Education Indicators, Council of Chief 
State School Officers; Hilda Borko, School of Education, University of 
Colorado, Boulder; Robert Crangle, President, Rose & Crangle, Ltd., Lin- 
coln, KS; Daniel L. Goroff, Department of Mathematics; and Derek Bok 
Center for Teaching and Learning, Harvard University. 

Although the reviewers listed above have provided many constructive 
comments and suggestions, they were not asked to endorse the conclusions 
or recommendations, nor did they see the final draft of the report before its 
release. The review of this report was overseen by Norman Hackerman, 
Scientific Advisory Board, The Robert A. Welch Foundation, Houston, TX 
and Department of Chemistry and Biochemistry (emeritus). The Univer- 
sity of Texas at Austin. Appointed by the NRC, he was responsible for 
making certain that an independent examination of this report was carried 
out in accordance with institutional procedures and that all review com- 
ments were carefully considered. Responsibility for the final content of this 
report rests entirely with the authoring committee and the institution. 

Lauress L. Wise, Chair 
Lisa Towne, Study Director 
Committee on Research in Education 
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Executive Summary 



P eer review is a method used to inform decision making by engaging 
experts in a critical evaluation of the merits of a product or pro- 
posal. It is most commonly known as a mechanism for judging the 
quality of proposals for research funding, or manuscripts submitted for 
publication in academic journals. 

The focus of this report is on peer review as it is applied to the evalua- 
tion of proposals for federal funding of education research projects. The 
primary source of evidence we use to set forth our conclusions and recom- 
mendations about this topic is a workshop we convened in February 2003 
in Washington, DC. The agenda for that event, a full transcript of the day’s 
presentations and discussions, and a paper we commissioned to lay the 
groundwork for our deliberations are all available on the web at http:// 
www7.nationalacademies.org/core/Peer%20Review.html. 

A long-standing tool of science policy in the United States, peer re- 
view is widely recognized as the preferred method for judging the merits of 
proposals for research funding. Across the federal government, it is used in 
a variety of contexts and for a variety of purposes — both scientific and 
political in nature. It is at once a tool with which scientific judgment is 
formalized and decisions about the allocation of scarce public resources are 
legitimized. 



1 
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2 STRENGTHENING PEER REVIEW 

In the many federal agencies that support education research , 1 peer 
review practices vary widely. Historically, this variation has largely been a 
function of differences in culture, tradition, mission, and the fields and 
disciplines each agency supports. These factors are important inputs to the 
development of peer review systems, and each agency should have the flex- 
ibility to create a system tailored to meet its needs and to accommodate its 
circumstances. While we acknowledge this variation and encourage agency 
flexibility, all peer review systems can and should uphold basic principles of 
fairness and merit. 

For federal agencies that support education research, we recommend 
that in addition to these considerations, peer review should be explicitly 
designed, managed, and evaluated to promote two key objectives: to iden- 
tify and support high-quality scientific research in education and to pro- 
mote the professional development of the field. Adopting these objectives 
as guideposts will enable agencies to think strategically about the merits of, 
and trade-offs associated with, particular practices and design options. To 
illustrate this central idea, we analyze key elements of peer review through 
this lens. 

IDENTIFYING AND SUPPORTING HIGH-QUALITY RESEARCH 

Peer review can foster the development of a high-quality research port- 
folio in federal agencies that support education research. Two central issues 
form the core of how to develop a system that supports this goal: deciding 
who counts as a peer and defining and consistently implementing stan- 
dards of what high-quality means. 

Assembling the group of reviewers is the very crux of the matter: the 
peer review process, no matter how well designed, is only as effective as the 
people involved. Judging the competence of peers — individually and as a 
group — in any research field is a complex task requiring assessment on a 



’The agencies who made presentations at our workshop included the Institute of Edu- 
cation Sciences and the Office of Special Education Programs of the U.S. Department of 
Education; the Center for Scientific Review and the National Institute on Drug Abuse of the 
National Institutes of Health; the Education and Human Resources Directorate and the 
Social Behavioral and Economic Sciences Directorate of the National Science Foundation; 
and the Office of Naval Research. 



Copyright © National Academy of Sciences. All rights reserved. 



Strengthening Peer Review in Federal Agencies That Support Education Research 
http://www.nap.edU/catalog/1 1 042.html 



EXECUTIVE SUMMARY 3 

number of levels. In education research, it is particularly complex because 
the field is so diverse (e.g., with respect to disciplinary training and back- 
ground, epistemological orientation) and diffuse (e.g., housed in various 
university departments and research institutions, working on a wide range 
of education problems and issues). 

The types of expertise reviewers should bring to the peer review table 
to identify high-quality education research include the content areas of the 
proposed work; the theoretical models, methods, and analytic techniques 
proposed to address the research questions; and the practice and policy 
contexts in which the work is situated. No single individual is expected to 
contribute expertise in all of these areas. Rather, panels must bring com- 
bined expertise; thus, expertise in the context of assembling peer reviewers 
should be viewed in terms of the panel as a whole. 

Vetting and publicly disclosing reviewers’ potential conflicts of interest 
and biases is essential to identifying panelists who will fairly judge propos- 
als on their merits. In practice, many top-flight researchers predictably have 
potential conflicts or biases with respect to particular applicants in a re- 
search competition: they may have collaborated on projects, coauthored 
papers, or mentored or taught the investigator seeking funding. Reviewers 
must be as free as possible from such potential conflicts of interest, as they 
cast doubt on their ability to judge the proposals on their merit alone. 
However, because many of the best reviewers are likely to be linked in some 
way to the work under consideration, if conflict of interest rules are too 
strict, the talent pool of reviewers can shrink dramatically. Also, the biases, 
or preferences, that reviewers bring to their work should be aired and bal- 
anced across panelists so that no single paradigm or perspective dominates 
the panel review. The key is public disclosure of potential conflicts and 
biases so that others can gauge the significance of any identified conflict 
and bring up those that might not be identified. 

Two broad types of diversity are relevant to assembling high-quality 
panels and to promoting education research quality through peer review: 
diversity of scholarly perspectives and diversity of groups traditionally 
underrepresented in education research. Engaging researchers who ap- 
proach the topic under consideration from a range of perspectives can en- 
rich peer review deliberations by bringing together a diverse set of expertise 
around a common set of issues and problems. And explicitly reaching out 
to traditionally underrepresented populations to participate in peer review 
fosters an environment in which questions are provoked and issues raised 
that otherwise might not have surfaced, helping to ground the review in 
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4 STRENGTHENING PEER REVIEW 

the cultural and social contexts in which the work is proposed to be con- 
ducted and promoting the quality of the research over time. Actively pursu- 
ing diversity along both of these dimensions in an agency’s peer review 
system has other benefits as well, including lending the process legitimacy 
and enhancing and extending powerful learning opportunities that take 
place in peer review deliberations. 

There is a final and particularly contentious issue related to diversity 
and to identifying the peers that review in federal agencies that support 
education research: how education practitioners and community members 
should be involved. Because education research is applied and attention to 
the relevance of the work crucial, it is essential to include practitioners and 
community members in the work of such agencies. Whether and how they 
participate on panels, however, is a difficult question. A major concern 
with the practice of including reviewers without research expertise 2 on pan- 
els is that it could lead to inadequate reviews with respect to technical merit 
criteria, a critical aspect of research proposal review in all agencies. In addi- 
tion, since the field of education research is in the early stages of developing 
scientific norms for peer review, this important process could be compli- 
cated or slowed by the participation of individuals who do not have a back- 
ground in research. We do see the potential benefits of including practitio- 
ners and community members on panels evaluating education research 
funding applications for identifying high-quality proposals and contribut- 
ing to professional development opportunities for researchers, practitio- 
ners, and community members alike. Thus, we conclude that this option is 
one of four possible strategies — including reviewing proposals alongside 
researchers, reviewing proposals after researchers’ reviews, serving on prior- 
ity setting or policy boards, or participating in retrospective reviews of 
agency portfolios — that agencies could adopt to actively engage practitio- 
ner and community members groups in their work. 

To promote high-quality education research that is rigorous and rel- 
evant, peer review must rest on consistently applied and well-defined qual- 
ity standards, and these standards should be made clear to applicants and to 
reviewers. Although the specific criteria by which applications are assessed 



2 We recognize that some practitioners and community members do have research ex- 
pertise. In these cases, the concerns we outline do not apply. Our focus here is on those 
practitioners and community members who do not bring this expertise to peer review delib- 
erations. 
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EXECUTIVE SUMMARY 5 

in the review process can and should vary by agency, it is important that 
each develop specific scales and anchors for reviewers to apply to each evalu- 
ation criterion to promote a fair and reliable process. Furthermore, it is 
essential that reviewers be trained in how to apply the review criteria prior 
to their evaluation of proposals, so all reviewers approach the task with a set 
of common understandings about evaluative criteria. 

Peer reviewers provide important input to agency leaders about what 
research applications should be funded. However, it is most often the case 
that staff is authorized to make final funding decisions. Thus, internal deci- 
sion making should also be driven by quality considerations. Simply going 
down a list of applications that peer reviewers have ranked in terms of 
quality and funding them until available dollars are depleted, for example, 
can lower the quality of the portfolio as a whole if the quality of the propos- 
als drops off at place that is above (or below) the point at which funding 
runs out. 

Finally, risk is an important element of a high-quality education re- 
search portfolio: if agencies never support new work that strikes off in a 
new direction, challenges core ideas, or approaches a problem from a novel 
perspective, the potential for significant progress, or even breakthroughs, 
will be substantially curtailed. The value of risk-taking and innovation 
should be reflected in funding decisions. This can be accomplished in a 
number of ways: by including “innovation” or a related construct in the 
evaluation criteria for the reviews, by allowing agency decision makers to 
fund applications “out of order” to support riskier proposals, or by retain- 
ing a funding mechanism outside the peer review process that is designed 
to support highly innovative work. 

FURTHER DEVELOPING A 
PROFESSIONAL CULTURE OF INQUIRY 

Peer review in federal agencies is rarely designed with the intent of 
using the tool to enhance the capacity of the field to support a culture of 
rigorous inquiry. We view this objective as crucial and recommend that 
peer review systems be designed to support its attainment 
explicitly. Practices that can support this goal include the targeted inclusion 
of panelists from a range of scholarly perspectives and traditionally 
underrepresented groups in education research (note that this strategy pro- 
motes this professional development goal as well as the goal of identifying 
and supporting high-quality research), the use of standing panels, the con- 
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6 STRENGTHENING PEER REVIEW 

sistent provision of comprehensive feedback to applicants, the development 
of agency staff, and extending opportunities for training and professional 
development among applicants, reviewers, and staff. 

The act of deliberating on the merits of research proposals can be a 
powerful learning opportunity for everyone involved. To encourage the use 
of peer review as a tool for raising the capacity of the field as a whole, 
agencies should expand their pool of reviewers by reaching out to groups 
traditionally underrepresented in education research. Similarly, finding ways 
to support the participation of junior scholars in the peer review process 
can be an effective mechanism for mentoring the next generation of educa- 
tion researchers. 

Standing panels in which scholars from a range of disciplines and per- 
spectives meet regularly to review research in a given area can offer a rich 
setting for integrating knowledge across domains and fostering ongoing 
learning opportunities for participants. They also offer the kind of stability 
and institutional knowledge that can facilitate positive outcomes when in- 
vestigators resubmit an application previously reviewed under their aus- 
pices. Standing panels do have their drawbacks, however; they can institu- 
tionalize bias and narrow the kinds of expertise that can and should be 
brought to bear on peer review deliberations. Their use should be designed 
to offset such weaknesses, through the use of such practices as staggered 
terms for members. 

The professional development of applicants — both successful and un- 
successful — can be enhanced through the provision of consistent, compre- 
hensive feedback on their proposals. In this way, the considerations and 
perspectives of the range of expertise and experiences represented in the 
peer review process can be communicated to applicants, enhancing the 
likelihood that their future work will be strengthened. Similarly, interac- 
tions between agency staff and prospective applicants can serve as a com- 
munication channel between these two key groups in education research. 

Since agency staff are important actors in the education research com- 
munity generally and in the peer review process specifically, their role re- 
quires careful consideration. Authorizing staff to be substantively involved 
in all aspects of research competitions — from writing requests for proposals 
to running review panels — can draw on and strengthen their expertise and 
provide useful continuity throughout the process. However, it can also raise 
thorny questions about fairness. If the same staff work with potential appli- 
cants, select reviewers, and play important roles in final decisions about 
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funding, it can create at least the perception that researchers who know 
agency staff will have an unfair advantage over their less connected peers. 
Federal agencies that support education research should balance the provi- 
sion of professional development opportunities for staff and the effective 
use of their expertise in the process against the need to ensure fair reviews 
and a system that is viewed as legitimate among a range of stakeholders. 

Finally, agencies can enhance the use of peer review as a tool for ex- 
panding the capacity of the field by providing training to those involved in 
the peer review process and offering professional development opportuni- 
ties for staff and a range of field researchers. Agencies should share this 
responsibility for making high-quality training widely available with rel- 
evant scientific and professional associations. 

AGENCY MANAGEMENT AND INFRASTRUCTURE 

Effective organizational practices and strong infrastructures are essen- 
tial to any well-functioning peer review system, including those supported 
by federal agencies that fund education research. Organizational excellence 
should be supported by a focus on evaluation, and agencies should consis- 
tently analyze the extent to which their practices support stated goals. In 
addition, effective management requires planning and organization in ad- 
vance of a review. Scheduling reviews with appropriate amounts of lead 
time in federal agencies that support education research has been hampered 
in the past by unpredictable timing of appropriations and widely fluctuat- 
ing levels of funding. Internal barriers within agencies that slow down pro- 
gram announcements, make peer review difficult to schedule, and result in 
complicated or burdensome logistics should be minimized. Similarly, legis- 
lative mandates that prescribe the details of peer review systems should be 
minimized, as they can hinder the development of quality systems and 
impede progress. 

An agency infrastructure built to support peer review systems should 
include, at a minimum, knowledgeable staff, systems for managing the lo- 
gistics of peer review, and the strategic use of information technologies to 
support review and discussion of proposals. In particular, we suggest agen- 
cies invest in the development of databases that house detailed information 
on past and prospective reviewers to facilitate the identification of high- 
quality peers. 
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CONCLUSION 

Peer review has been held up as a quality standard for the conduct and 
use of education research. It is clear that understanding the complexities 
and trade-offs associated with this tool is required for the standard to be 
applied consistently and well. We offer this brief treatment to encourage its 
use as a tool for promoting important objectives in the improvement of 
education research and to provide a framework for research policy makers 
charged with overseeing peer review systems designed to assess proposals 
for education research funding. Despite its flaws, peer review is a system 
worth preserving and improving. It can be a powerful driver for the im- 
provement of education research and the field — provided that those charged 
with overseeing the processes understand the strengths and weaknesses of 
various approaches and implement them with clarity of purpose. 
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Setting the Stage 



P eer review is a method used to inform decision making by engaging 
experts in a critical evaluation of the merits of a product or pro- 
posal. Although it takes on many forms and serves a variety of pur- 
poses, it is most commonly known as a mechanism for judging the quality 
of proposals for research funding or manuscripts submitted for publication 
in academic journals. 

The focus of this report is on peer review as it is applied to the evalua- 
tion of proposals for federal funding of education research projects. To set 
the proper context for our treatment of this topic, this chapter provides an 
overview of the nature and use of peer review in U.S. science policy; out- 
lines current policies and initiatives with implications for peer review in 
federal agencies; highlights the inherent tensions between political and sci- 
entific values in federal peer review systems (in education in particular), 
and describes our sources of evidence for, and approach to, setting forth 
our conclusions and recommendations. 

A TOOL OF U.S. SCIENCE POLICY 

Although there is no single definition of peer review across the many 
federal agencies that employ it (U.S. General Accounting Office, 1999), it 
is essentially a mechanism by which experts (“peers”) provide input to deci- 
sion makers on the merits of proposals for research dollars. Peer review 
confers the imprimatur of high-quality to research proposals throughout 

9 
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the federal government — judging quality (and potential quality) and lend- 
ing credence to the allocation of scarce public resources. 

The use of peer review in the United States can be traced back to the 
19th century; for example, the Smithsonian Institution created an advisory 
committee for reviewing and recommending funding proposals in the 
1840s (Guston, 2000). Since the 1940s and 1950s, peer review has evolved 
into a cornerstone of federal science policy, and today it is widely recog- 
nized as the preferred method for judging the merits of research proposals 
(Kostoff, 1994), for the ultimate purpose of improving government deci- 
sion making. More specifically (Guston, 2000, pp. 4-31): 

Reformers have sought to harness peer review to help produce 
knowledge on which policy makers can rely, for the ultimate pur- 
poses of improving decisions, reducing the occurrence of legal 
challenges and other procedural obstacles, and achieving other 
political goals. 

As a tool of science policy, peer review is embedded in the American 
system of government. Since it is used to steer the investment of federal 
funding in research, peer review is necessarily and appropriately influenced 
by political values and accountable to elected officials for the fairness of its 
processes and the success of its outcomes. Accountability to the public sug- 
gests, among other things, that the process be transparent and legitimate to 
a broad range of stakeholders. 

As a tool for researchers to decide the merits of new research projects 
and directions, peer review is also part of the fabric of scientific communi- 
ties. 1 Scientific judgment is formalized through the peer review process, as 
groups of researchers familiar with an area of scientific inquiry reflect on 
the likelihood of proposed ideas to advance what is known in productive 
ways. As such, peer review is a tool through which researchers develop, 
sustain, and communicate their professional culture, suggesting the need 
for a buffer from political influences. 

Peer review in federal agencies, then, reflects both the principles of 
democratic accountability and the principles of scientific merit. In practice, 
upholding both sets of principles simultaneously can be difficult. The ways 
in which agencies go about promoting both take on many different forms. 



1 Although not our focus, peer review is also a hallmark of research funding decision 
making in the arts and humanities. 
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and negotiating the proper roles and boundaries of each is an inescapable 
and constant task, as the description of the current policy context below 
makes clear. 



POLICY CONTEXT 

No single policy sets standards for, or provides oversight of the use of, 
peer review in the federal government. In fact, peer review is largely a prac- 
tice shaped by culture and experience. However, there are a host of statutes, 
regulations, and other policy issuances that influence (or could influence) 
the structure and use of peer review across a broad swath of agencies, in- 
cluding those that fund education research. Some of these policies invoke 
peer review as a way to promote particular objectives, including enhancing 
the technical quality and credibility of information disseminated by federal 
agencies (e.g., U.S. Office of Management and Budget, 2003, 2004), sup- 
porting research that is rigorous and relevant to national problems (e.g.. 
Education Sciences Reform Act of 2002, H.R. 3801), and using scientific 
advisory panels to inform government decisions of many kinds (and thus is 
subject to the Federal Advisory Committee Act, EL. 92-463, Sec. 1, Oct. 
1972). In addition, there are policies not about peer review per se that 
nonetheless have potentially significant implications for its management 
and use. For example, the President’s Management Agenda, a recent direc- 
tive aimed at managing and investing federal dollars more efficiently, has 
as-yet unknown implications for federal personnel, including those who 
oversee the peer review infrastructure in scientific agencies. 

In the past few years, federal officials have set forth many of these new 
initiatives and proposed a range of clarifications and modifications to other 
elements of this policy apparatus, inviting a spirited debate and illustrating 
the tensions that arise as peer review serves both political and scientific 
ends. We provide an overview of a few of them here to characterize the 
current landscape of which our consideration of peer review of education 
research proposals is a part, to demonstrate the high stakes associated with 
the topic, and to illustrate and foreshadow some of the complexities of peer 
review in a range of fields, disciplines, and applications. 

For example, the U.S. Office of Management and Budget (OMB) — 
the agency in the Executive Office of the President charged with develop- 
ing and overseeing the implementation of the federal budget — recently is- 
sued a bulletin to provide guidance for defining government-wide standards 
for the peer review of “influential” scientific information and assessment 
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(including social science studies). The intent of the guidance is to improve 
the quality of the information the government uses to make policy, for 
example, in developing federal regulations that govern the nation’s safety, 
health, and environmental policies (U.S. Office of Management and Bud- 
get, 2003, 2004). Although the purview of the bulletin explicitly excludes 
the peer review of research proposals (our specific topic), the input of the 
scientific communities critiquing an early draft of the bulletin raised im- 
portant questions and issues about the roles and purposes of peer review in 
general, many of which we address in this report. Later in this chapter, we 
highlight some of these commonalities as prologue to our consideration of 
peer review of education research proposals in federal agencies. 

Another recent example of policy reforms with implications for peer 
review stems from the competitive sourcing and consolidation directives 
contained in the Presidents Management Agenda, the Federal Activities 
Inventory Reform Act of 1998, P.L. 103-270, andtheOMB Circular Num- 
ber A -76. Overall, these initiatives are intended to promote efficiency in 
government operations. Agency implementation of these wide-ranging ini- 
tiatives features, among other requirements, a multiyear effort to catalogue 
and to assess whether the functions performed by federal personnel — in- 
cluding those who support the peer review infrastructure in research agen- 
cies — are “inherently governmental functions” and whether they could be 
more efficiently performed by private sources. While the scope of this ef- 
fort and its implications for federal jobs are still in flux, it is possible that 
organizations like the National Institutes of Health (NIH) Center for Sci- 
entific Review (which oversees the peer review of grant proposals to the 
agency) could be downsized. Because peer review is so central to NIH and 
other research agencies, many researchers have expressed concern about the 
potential negative consequences of this initiative for the peer review infra- 
structure in the federal government (Kaiser, 2003). 

The centrality of peer review for funding decisions in scientific agen- 
cies and for the use of scientific information throughout the government is 
also reflected in recent agency planning documents prepared to comply 
with the Government Performance and Results Act of 1993, P.L. 103-62. 
The Department of Interior’s most recent strategic plan, for example, states: 
“The world-wide hallmark of good science is the collegial, cooperative, 
peer review of study plans and experimental results.” Similarly, the U.S. 
Department of Education’s (ED) strategic plan cites the need to “articulate 
clear standards” for peer review as a strategy for raising the quality of re- 
search funded or conducted by the department. 
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The concerns raised by scientists in response to the OMB peer review 
bulletin and the competitive sourcing efforts, coupled with the prominence 
of peer review in strategic planning documents in federal agencies, suggest 
that there is strong consensus about the important role of peer review 
among researchers and policy makers alike. But because it sits at the nexus 
of science and government, peer review is not without its controversies in 
practice. Recent events in the halls of Congress provide another case in 
point. During debate on the fiscal year 2004 labor, health and human ser- 
vices, and education appropriations bill, an amendment was introduced to 
eliminate funding for a handful of projects that had passed through the 
NIH peer review system because the subject of inquiry — human sexual- 
ity — was objectionable to some members. A debate ensued about the ap- 
propriate role of Congress vis-a-vis the merit-based peer review process. In 
the end, the amendment was defeated and the projects maintained fund- 
ing, but the controversy continues. 

Recent federal initiatives in education research and the critical role of 
peer review in it also illustrate the often controversial negotiations and high- 
stakes debates that characterize peer review in the federal government, pro- 
viding further context for our consideration of peer review with respect to 
education research funding in the federal government. Most notable in this 
regard are a set of federal education policies aimed at applying “scientifi- 
cally based research” to improve policy and practice. In many recent reau- 
thorizations of the K-12 education laws that govern the federal role in el- 
ementary and secondary education, there is explicit reference to the use of 
research to inform reform efforts. Such language appears over 100 times in 
the No Child Left Behind Act of 2001 alone. A component of the detailed 
definition of scientifically based research in the act — which is essentially 
the standard that federal grantees must meet in providing evidence to sup- 
port their program choices — is that it “has been accepted by a peer-re- 
viewed journal or approved by a panel of independent experts through a 
comparably rigorous, objective, and scientific review” (P.L. 107-110, the 
No Child Left Behind Act). Similar provisions that include reference to 
peer review appear in the Education Sciences Reform Act, H.R. 3801, leg- 
islation passed in 2002 that replaced the Office of Educational Research 
and Improvement (OERI) with the Institute of Education Sciences (IES) 
as the primary research arm of the ED. 

In crafting these provisions, members of Congress and their staffs 
aimed to upgrade the quality of research and to promote its widespread use 
(see, e.g., Sweet, 2002); here again is a reflection of policy makers’ faith in 
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peer review. Among many education researchers, however, the insertion of 
these provisions into federal law has provoked protests about what they 
view as an encroachment on their profession to define legislatively (politi- 
cally) what “counts” as rigorous, scientific education research (Feuer, Towne, 
and Shavelson, 2002; Erickson and Gutierrez, 2002). Arguably, the profile 
of education research is at a high point, but so too are the controversies in 
and around the field of education researchers. 

In this environment, IES officials have recently made changes to their 
peer review system. The leadership of the agency contracted out the man- 
agement of their peer review system, a responsibility formerly held by 
agency staff. IES director Grover (Russ) Whitehurst explained this decision 
at the workshop, saying that it was made to reduce the likelihood of bias or 
the appearance of bias in the agency’s peer review system by separating 
agency staff who develop research program announcements from the re- 
view process. In the current system, IES staff generates lists of potential 
reviewers. In conjunction with the chair of each review panel, the indepen- 
dent contractor then vets and appoints the reviewers, with Whitehurst ap- 
proving the final list. Concern was raised by participants at the workshop 
and elsewhere that this arrangement could result in the exclusion of rel- 
evant perspectives from the important considerations that take place at the 
peer review table. 

We do not intend to adjudicate the current or future consequences — 
positive or negative — that may come of any of the initiatives we highlight 
in this brief sketch of the policy landscape. In addition to providing context 
for our consideration of peer review of education research proposals, this 
snapshot of recent events illustrates the complexities inherent in peer re- 
view in general and relevant activities in education research specifically, 
laying the foundation for our discussion of the use of peer review in federal 
agencies that support education research. 

IMPLICATIONS AND THEMES 

Taken together, the conversations that have swirled around this array 
of activity go to the core issues involved in developing effective peer review 
systems and the proper role of researchers, elected officials, and other stake- 
holders in it. Indeed, many of the same issues are raised regardless of the 
type of review or the scholarly field in which the debate is situated. For 
example, a November 2003 National Research Council (NRC) workshop 
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convened by the Science, Technology, and Law Program on Peer Review 
Standards for Regulatory Science and Technical Information convened 
stakeholders to discuss the initial draft of the OMB peer review bulletin. In 
offering a conceptual overview of issues in this context, Sheila Jasanoff, a 
leading expert in science policy and author of the influential 1990 book 
The Fifth Branch, noted that peer review serves a number of different pur- 
poses in government and that its processes vary substantially across con- 
texts, citing the “endless diversity” of existing models and insisting on the 
need for flexibility in agencies to develop systems that meet their needs. 
John Graham, the OMB official leading the development and implementa- 
tion of the bulletin, stressed the need to bring top experts to the table, but 
concerns were raised about who to involve and how to deal with potential 
conflicts of interest and biases. Donald Kennedy, editor-in-chief of Science 
magazine, discussed the importance of, and complexities involved in, de- 
veloping a process that is sufficiently transparent and inclusive to promote 
its legitimacy among a range of stakeholder groups, including practitioners 
and consumers. 

These issues were central to the discussion of education research peer 
review at our own workshop and are prominent themes in this report. Echo- 
ing JasanofFs observation of the multiple purposes that peer review is asked 
to fulfill and the variation in models across and within agencies that fund 
education research, we argue for flexibility for agencies coupled with a clear 
articulation of the objectives each system is designed to serve. We tackle the 
tough question of what counts as expertise in the diverse and applied field 
of education research. Specific issues of whether and how to involve stake- 
holders in the process and how to identify and deal with potential conflicts 
of interest and bias among candidates for participating in peer review pan- 
els are also addressed, picking up on the issues Graham and members of the 
scientific community raised about assembling panelists. And finally, a theme 
that runs throughout the report echoes Kennedy’s concern of ensuring a 
system that is widely viewed as legitimate by the many diverse stakeholders 
in science, including scientific education research. 

SOURCES OF EVIDENCE 

This report is one in a series designed to highlight and to promote 
improvements in the quality of scientific research in education. Our con- 
sideration of peer review as one important leverage point for promoting 
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quality is informed by three main sources of evidence: the workshop the 
committee held on the topic in February 2003, a select review of relevant 
literature, and our own experience on peer review panels. 

We designed the workshop to promote a broad-based discussion of the 
purposes, models, and results of peer review systems used across a range of 
federal agencies that support education research in light of the recent 
changes and issues in the policy landscape we have described. To help frame 
the event, we commissioned Edward Hackett and Daryl Chubin, authors 
of the highly influential book Peerless Science: Peer Review and U.S. Science 
Policy (Chubin and Hackett, 1990), to write a paper on the institutional 
and social contexts of peer review of education research proposals and to 
present major themes at the start of the workshop. We organized the rest of 
the workshop sessions to address topics and to feature speakers that pro- 
vided maximum coverage of issues and representation of viewpoints. Work- 
shop speakers included federal officials from the ED; the NIH; the Office 
of Naval Research (ONR); and the National Science Foundation (NSF). In 
addition, speakers included investigators from a range of fields in educa- 
tion research and social science disciplines as well as from epidemiology 
and nuclear physics. 

Over the course of the day and a half long dialogue, this diverse set of 
experts and policy makers provided their perspectives and experiences and, 
in limited cases, the results of their research, on such issues as the purposes 
of reviews, selection and training of reviewers, review criteria and scoring 
systems, roles of staff and stakeholders in the process, and timelines and 
other management issues. The agenda for the meeting, a transcript of the 
presentations and discussion sessions, and the Hackett and Chubin paper 
can be found at http://www7.nationalacademies.org/core/Peer%20 
Review.html. The wide-ranging and probing dialogue that resulted from 
the paper and the event are the main sources of support for the conclu- 
sions and recommendations in this report. 

A secondary source of evidence for our conclusions and recommenda- 
tions is a select review of the published literature on peer review. This em- 
pirical research base on peer review of research proposals is surprisingly 
limited; while there are some important exceptions (several of which we 
cite in this report, and some of which we individually authored), most 
publications on the topic are based on theoretical arguments and personal 
experience. We draw most heavily from a few seminal publications that 
provide a survey treatment of issues in peer review and that focus on educa- 
tion research specifically, including Chubin and Hackett (1990), the U.S. 
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General Accounting Office (1999), August and Muraskin (1998), and sev- 
eral publications of the NRC. The NRC reports we consulted related to 
peer review and science policy include Peer Review in Environmental Tech- 
nology Development Programs (National Research Council, 1999), Evaluat- 
ing Federal Research Programs: Research and the Government Performance and 
Results Act (Committee on Science, Engineering, and Public Policy, 1999), 
and Assessing the Need for Independent Project Reviews in the Department of 
Energy (National Research Council, 1998). We also relied on NRC reports 
related to education research, including Research and Education Reform: 
Roles for the Office of Educational Research and Improvement (National Re- 
search Council, 1992) and Scientific Research in Education (National Re- 
search Council, 2002). 

Finally, we draw on our own personal experience serving on panels to 
supplement the evidence gathered during the workshop and in our limited 
review of published literature. Our collective experience on panels is sub- 
stantial, including service for such federal agencies as the IES and its prede- 
cessor agencies, several directorates of the NSF, a number of study sections 
in the NIH, the Veteran’s Administration, the Department of Defense, and 
the Department of Energy; state agencies, such as the Massachusetts De- 
partment of Health; participation in panel reviews of research proposals to 
be funded by other national governments, such as those in the United King- 
dom, Canada, Japan, Australia, and Israel; research reviews for philan- 
thropic foundations and professional associations, such as the American 
Educational Research Association, the Association of Teachers of Preventa- 
tive Medicine, the March of Dimes, the Spencer Foundation, the Markle 
Foundation, the Russell Sage Foundation, the Sloan Foundation, the All 
Kinds of Minds Foundation, and the McDonnell Foundation; and in peer 
review deliberations across a range of disciplines and fields, including soci- 
ology, history, psychology, cognition, epidemiology, statistics, cultural an- 
thropology, nuclear physics, and an array of subfields in education research. 



OBJECTIVE AND APPROACH 

Our main objective in issuing this report is to inform decision makers 
charged with developing or maintaining peer review systems for education 
research proposals in a rapidly changing policy context. To meet this objec- 
tive, we set a number of parameters to frame how we would approach 
designing the workshop and developing this report. We articulate these 
parameters to further orient the remaining discussion. 
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First, our conclusions and recommendations apply to peer review of 
education research proposals in any federal agency that funds research rel- 
evant to education. Our primary sponsor for the event — the National Edu- 
cational Research Policy and Priorities Board — was disbanded with the pas- 
sage of the Education Sciences Reform Act, H.R. 3801, in late 2002. That 
act created the IES, replacing the OERI and calling for the formation of a 
new policy board that will work with IES leaders on issues related to peer 
review in the agency. (As of this writing, the president has nominated sev- 
eral individuals to be members of the board; they now await confirmation 
by the Senate, as required by law.) In addition to informing this new board’s 
work, we also hope to provide direction for other government agencies that 
fund education research. Thus, we designed the workshop and wrote this 
report to apply to the range of federal agencies that use peer review (or will 
in the future) to aid education research funding decisions — including, but 
not isolated to, the IES. 

Furthermore, although the committee did hear about an evaluation of 
OERI’s peer review process in the 1990s, the workshop and this report are 
not evaluations of OERI or any other agency’s peer review system. Rather, 
we treated the information we learned about agency systems at our work- 
shop as examples of peer review in practice. We describe aspects of these 
systems, highlighting their strengths and weaknesses in meeting particular 
objectives and drawing on the experiences of the agency representatives to 
ground our deliberations and recommendations in the reality of the current 
federal policy environment. 

Finally, our treatment of peer review in this report focuses only on 
federal systems designed to handle proposals for education research projects. 
The use of peers to judge proposals or products is common in other pur- 
suits both inside and outside government. In the ED, for example, peer 
review has been used to judge state accountability plans for implementing 
certain provisions in Title I of the No Child Left Behind Act, as well as to 
provide input on state applications for funding under the discretionary 
programs authorized by the law (e.g., Reading First). In nonprofit and pri- 
vate institutions with research programs (e.g., foundations), peer review is 
also frequently used. We expect that much of our discussion of the issues 
and our recommendations will be relevant to these other uses of peer re- 
view, but we have not analyzed their generalizability to other such areas 
explicitly. 

One other part of the education research infrastructure in which peer 
review plays an important role is in manuscript submissions to journals. 
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We did not take up this kind of peer review at our workshop in any detail 
and thus do not treat it in this report; we addressed this issue more directly 
in a later workshop on journals and their role in advancing knowledge in 
education (the workshop agenda and transcript from this event can be 
viewed at http:/ /www7. nationalacademies.org/ core/Journal%20Standards. 
html) . 



ORGANIZATION OF REPORT 

To develop a concise report for policy audiences, we treat a complex 
topic with over 100 years of history and scholarship thematically and selec- 
tively. We organize the remainder of this report into two chapters. Chapter 
2 provides an analysis of some of the key features of and debates around the 
peer review of education research proposals. Building on that discussion. 
Chapter 3 contains our proposals for strengthening and improving peer 
review systems used to aid decision making about the federal funding of 
education research. 
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Analyzing Key Elements 



O n the surface, setting up an effective peer review system seems 
straightforward: contact the top experts in the field, have them 
review a set of proposals and provide their input, and fund re- 
search projects according to the advice provided. Experience has shown, 
however, that enacting this simple concept is complicated. How should 
expertise be defined, especially in the many areas of education research that 
are multidisciplinary? What kind of criteria should be used to judge the 
proposals, and how should they be quantified or summarized? How should 
the process be structured so it is seen as legitimate by a range of stakehold- 
ers? What is the best way to organize and support the group? What is the 
nature of the relationship between the peers and the agency staff and lead- 
ership, who typically make, and are ultimately accountable for, final fund- 
ing decisions? 

These are but a few of the multidimensional questions involved in 
designing, revamping, or evaluating peer review systems. This chapter pro- 
vides an overview of some of the major components of peer review systems 
designed to assess education research proposals in federal agencies. We de- 
scribe and analyze components of peer review processes with respect to how 
they promote particular objectives. We chose to consider aspects of peer 
review from this perspective not only because it serves as an effective orga- 
nizational framework, but also because in our view research policy makers 
ought to approach their own systems in a similar manner. We conclude 
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with an examination of management issues that influence the extent to 
which such systems can produce desired results. 

From our analysis, we draw six major conclusions: 

1. Peer review serves a number of worthwhile purposes. For peer re- 
view systems for federally funded education research, two objectives 
important in their design are the identification and support of high- 
quality research and the further development of a culture of rigor- 
ous inquiry in the field. 

2. Federal agencies that fund education research use a range of models 
for peer review that serve different purposes and objectives. 

3. Developing peer review systems involves balancing multiple, 
and sometimes conflicting, values and thus often requires making 
trade-offs. 

4. Peer review in the federal government is a tool by which agency 
goals are accomplished and therefore can only be developed, evalu- 
ated, and understood as framed by these objectives. 

5. Although peer review is not perfect, it is the best available mecha- 
nism for identifying and supporting high-quality research. 

6. Peer review of education research proposals in federal agencies could 
be improved in a number of ways. 

MULTIPLE PURPOSES AND VALUES 

In Chapter 1, we described the nature of peer review in federal govern- 
ment as one that serves both scientific and political ends. 

In their paper (see http://www7.nationalacademies.org/core/Hacket 
Chubin_peer_review_paper.pdf) and presentation to the committee, 
Hackett and Chubin elaborate the many functions that peer review is called 
on to serve. At the most basic level, peer review is a mechanism for evaluat- 
ing the merits of proposals for research funding, thereby influencing the 
distribution of federal research funds. But it also serves several additional 
and related functions. 

For example, a major reason scientists participate in peer review — a 
time-consuming task in addition to existing professional obligations — is to 
have an impact on the field beyond their own investigations. Thus, peer 
review shapes the accumulation of knowledge over time by recommending 
a subset of proposed research for implementation. This idea was prominent 
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in workshop discussions. Both Hilda Borko, education professor at Univer- 
sity of Colorado and president of the American Educational Research Asso- 
ciation, and Penelope Peterson, dean, school of education and social policy 
and Eleanor R. Baldwin Professor of Education at Northwestern Univer- 
sity, speaking on behalf of a group of education school deans, 1 articulated 
peer review as a force that “shape[s] and envisions” the future of a field. 
Edward Hackett, sociology professor at Arizona State University highlighted 
the “communication function” of peer review and its role in “preparing] 
the ground for the acceptance of new ideas.” Finbarr Sloane, of the Educa- 
tion and Human Resources (EHR) Directorate of the National Science 
Foundation (NSF), echoed these ideas, stating that “there is a huge return 
on investment for serving on a panel. . . . [Reviewers] get a sense ... for 
what national questions other people are posing, and responses to those 
questions.” And Edward Redish — a physicist and physics education re- 
searcher at the University of Maryland — also pointed to the benefits for 
researchers who serve on peer review panels, citing the value he has experi- 
enced in “see[ing] what people were thinking about in the field.” 

Delivering feedback to proposers can also signal the field’s (often im- 
plicit) standards of quality, reinforcing them in a formal context. Redish 
made this point about the purpose of peer review most directly, arguing 
that “peer review is not just about finding scientific merit in particular 
areas. It is about defining it and creating it.” This purpose is particularly 
salient in education, since current standards of evidence often vary by disci- 
pline and subfield. Redish’s point also underscores the fact that judging the 
scientific merit of a proposal for research is different from judging the mer- 
its of a research product. Research is by its nature an exercise in being alert 
to, and systematically dealing with, unexpected issues and questions that 
arise in the course of an investigation. Therefore, the nature and level of 
specificity of quality criteria are different when considering a description of 
how an investigator plans to approach the work than when considering the 
product of a completed investigation. 

Peer review can also be used as a tool for building interdisciplinary 



] This group — called the Education Deans’ Alliance — -was formed in 2000 to share 
information and to improve the doctoral training of education researchers at their institu- 
tions. It includes deans from schools of education at Columbia University, Emory University, 
Harvard University, Michigan State University, Northwestern University, Stanford Univer- 
sity, the University of California at Berkeley, the University of California at Los Angeles, the 
University of Michigan, and the University of Pennsylvania. 
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trust among groups of investigators from different research traditions — 
again, an important endeavor in an area like education, in which multiple 
fields and disciplines focus on various aspects of teaching, learning, and 
schooling. Kenneth Dodge, director of the Center for Child and Family 
Policy at Duke University, described how engaging in peer review helps 
draw disparate fields together to better reflect and understand the com- 
plexities of educational phenomena. 

Another function of peer review is its role as a buffer, creating a privi- 
leged space for researchers to make judgments largely apart from political 
considerations (Hackett and Chubin, 2003). While political considerations 
drive funding levels and can impact statements of priority areas (National 
Research Council, 2002), peer review is used to remove decisions about the 
funding of individual projects from the influence of special interests or 
other political groups and agendas. Thus, the peer review process offers a 
space for researchers to apply scientific principles, debate and identify prom- 
ising lines of inquiry, and offer crucial advice to decision makers that draws 
on their expertise to advance research-based knowledge. 

Workshop discussions also highlighted the role of peer review as a tool 
for professional development — for proposers, reviewers, and agency staff — 
to promote a professional culture of inquiry and rigor among researchers. 
This culture includes an ethos steeped in self-reflection and integrity, as 
well as a commitment to working toward shared standards of practice 
(Shulman, 1999; National Research Council, 2002; Feuer, Towne, and 
Shavelson, 2002). Many workshop participants pointed to the broad “edu- 
cative” function of peer review to mentor an incoming generation of schol- 
ars, to train investigators to review the scholarly quality of proposals, to 
produce higher quality proposals in the future, and to strengthen connec- 
tions throughout the field of education research. 

Although rarely explicit, peer review is often expected to meet these 
and many other purposes equally well. It is therefore not surprising that the 
process can come under fire for not serving any one of them fully. Design- 
ing peer review systems, improving existing ones, and assessing their effec- 
tiveness requires cognizance of these expectations and the implementation 
of process options accordingly. 

In addition to serving multiple purposes, peer review systems are also 
designed to serve a set of values, like those of the agency and the fields it 
supports. These values are sometimes in tension, and they always require a 
careful balancing act in choosing a course of action. For example, peer 
review is expected to uphold the value of effectiveness — “to recommend 
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projects that would benefit the field and confer some greater social benefit, 
to offer advice to proposers, to circulate ideas within a community, and 
more. Peer review is also asked to be efficient, to do all of this at very low 
cost, with cost measured in terms of dollars spent on reviews (infrastruc- 
ture, travel, reviewer compensation) and in hours expended by proposal 
writers and reviewers” (Hackett and Chubin, 2003, p. 15). 

Another example of these value tensions is the trade-off between risk 
and tradition. Hackett and Chubin (2003) argue that this tension in peer 
review is a reflection of the tension in scientific communities more gener- 
ally: research is expected to chart new progress, but to do so systematically 
and within the broad parameters set by existing knowledge and standards 
of rigor. During her presentation, Peterson argued that peer review sys- 
tems ought to “create opportunities for risk-taking and innovative educa- 
tion research.” 

Simultaneously maximizing efficiency and effectiveness and risk and 
tradition are just a few examples of the many kinds of values to be bal- 
anced — explicitly or implicitly — by peer review systems (see Hackett and 
Chubin, 2003, for a more complete treatment). The multiple purposes and 
competing values inherent to peer review, coupled with the complex nature 
of education and education research, are reflected in a high degree of vari- 
ability in peer review systems among the many agencies that fund educa- 
tion research. Culture, tradition, and the mission of the agency also exert a 
powerful influence over the nature of peer review practices. Indeed, it is 
clear that no single model could suit all purposes and all situations and all 
fields equally well. 

Whether a particular practice will work well depends in large part on 
the specifics of the situation and the purposes the system is intended to 
serve. To guide our analysis of peer review practices, we first articulate two 
broad purposes best served by peer review systems in federal agencies that 
support education research. 

KEY OBJECTIVES OF PEER REVIEW FOR 
EDUCATION RESEARCH 

Taking our cue from this discussion of multiple purposes, we conclude 
that two broad objectives that ought to guide the design of peer review 
systems in federal agencies: the identification and support of high-quality 
education research and the professional development of the field. 

The first objective of using peer review as a process to achieve quality 
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research has been front and center in federal agencies that have funded 
education research for some time (although it is a matter of debate how 
well various agencies have done so in the past). We strongly endorse explicit 
attention to education research quality as well as redoubled efforts to 
strengthen peer review systems for this purpose. Rigorous studies of educa- 
tional phenomena can provide important insights into policy and practice 
(and have — see National Research Council, 2002, for examples). But poor 
research is in many ways worse than no research at all, because it is wasteful 
and promotes flawed models for effective knowledge generation. Quality is 
of the essence, and having leaders in the field carefully scrutinizing and 
screening proposed work is one essential way to promote it. 

Although what is meant by quality with respect to education research 
is a matter of some debate in the field, attending to the rigor and relevance 
of education research is essential to its health. Peer review systems in federal 
agencies offer a natural place to engage the field in the contested but crucial 
task of developing and applying high standards for evaluating the merits of 
proposed research. Strict rules are not advisable given the interdisciplinary 
nature of education and the prospective nature of research proposals. How- 
ever, broad standards, consistently applied in peer review settings, are 
needed to ensure quality. 

Moreover, the current enthusiasm for, and debates surrounding, calls 
for “scientifically based research” in education and references to the use of 
peer review provide opportunities for a stronger and more consistent focus 
on peer review as the means to promote research quality. By defining and 
upholding high standards of quality in the peer review process, researchers 
can exert a powerful influence on questions of what counts as high-quality 
research in particular contexts — providing input directly from the scholarly 
communities with respect to the implementation of policies stemming from 
the now numerous definitions of quality research that appear in federal 
education law (e.g., the No Child Left Behind Act of 2001, the Education 
Sciences Reform Act of 2002, and bills pending to reauthorize the Indi- 
viduals with Disabilities Education Act of 1997 and parts of the Higher 
Education Act of 1965, P.L. 89-329). The insulation of peer review from 
the political process is important for facilitating this goal. 

In our view, the second objective that should guide peer review in 
federal agencies that support education research is to contribute to the fur- 
ther development of a culture of inquiry in the field. Peer review has not 
historically been designed to promote such professional development in 
the federal agencies that support education research. We think it deserves 
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far more attention. As the authors of Scientific Research in Education (Na- 
tional Research Council, 2002) argue, we think it is a professional responsi- 
bility of education researchers to participate in peer review in federal agen- 
cies, and the field ought to harness this system to promote the development 
of the profession. 

Federal education research policy makers also have major responsibil- 
ity for organizing peer review in ways that foster growth among education 
researchers. If deliberately developed with this objective in mind, peer re- 
view systems can serve this purpose among the many players in the educa- 
tion research field. In the context of peer review, they can usefully be cat- 
egorized as applicants (people who are seeking agency funds to initiate new 
work), reviewers (people who review the merits of the proposals for new 
work), and staff (people who work in the research agencies). 

All three of these categories of people are members of the research 
community, operating in the broader public domain. In the ideal, peer 
review systems foster enriching interactions, and each group serves both a 
teaching and learning function to their own benefit and that of others. 
Chubin and Hackett (1990) argue that this dynamic can improve under- 
standing among all members of the community, enhancing the capacity of 
the field as a whole. 

For example, an applicant can communicate to reviewers cutting-edge 
ideas in an area of study, stimulating thinking among a broader set of re- 
searchers on potential new directions for a field or subfield. In much the 
same way, the feedback that reviewers provide to applicants often signals 
areas of contention about new ideas or techniques, preparing the ground 
for broader scrutiny and consideration of where and how to push the knowl- 
edge base and its application. Agency staff teach and learn as well: they 
familiarize reviewers with relevant agency priorities, goals, review criteria, 
process specifics, and the particular objectives held in a research competi- 
tion for advancing the field. In the process of managing and participating 
in the process, the staff often gain a significant breadth of understanding 
and knowledge in a field by reading proposals and listening to reviewers’ 
dialogue about the status of the field and the quality of the batch of propos- 
als under review across and within panels. In some cases, agency staff are 
themselves accomplished researchers who are serving in temporary posts in 
research agencies. Overall, knowledgeable staff sharpen internal thinking 
about how to shape and run future competitions. 

Having described and justified our choice for the two objectives we 
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hold as most salient for shaping peer review of education research propos- 
als, we now analyze several design features of peer review systems described 
at the workshop with respect to how likely they are to promote them. Other 
purposes, including those mentioned in this report, may be relevant to 
promote in particular contexts and at particular points in the evolution of a 
line of inquiry in education research. Our intent in setting forth these two 
objectives is to identify explicitly the purposes we see as most relevant for 
organizing peer review systems in federal agencies, as well as to provide a 
structure for analyzing various aspects of peer review systems. Since some 
peer review practices serve more than one purpose, there is some overlap in 
the discussion of peer review practices and considerations between the two 
main sections that follow. In some of these cases, we highlight the tensions 
that arise and the trade-offs that are often required in the attempt of peer 
review to serve multiple purposes. 

IDENTIFYING AND SUPPORTING HIGH-QUALITY RESEARCH 

The formal review of education research proposals by professional peers 
must be designed to identify and support high-quality research. There are 
many decisions and practices that undergird this critical function, most of 
which can be categorized into two areas: the people in the process — Who 
counts as a peer? — and the criteria by which quality is judged — How is 
research quality defined? Within each, we take up a set of peer review prac- 
tices described at the workshop that relates to them most directly. 



Who Reviews: Identifying Peers 

Deciding who counts as a peer is the very crux of the matter: the peer 
review process, no matter how well designed, is only as good as the people 
involved. Judging the competence of peers in any research field is a com- 
plex task requiring assessment on a number of levels. In education research, 
it is particularly difficult because the field is so diverse (e.g., with respect to 
disciplinary training and background, epistemological orientation) and dif- 
fuse (e.g., housed in various university departments and research institu- 
tions, working on a wide range of education problems and issues). The 
workshop discussions brought out several related issues and illustrated the 
difficulties in, and disagreements associated with, assembling the right 
people for the job. 
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Unpacking Expertise 

What are the required skills, experiences, and knowledge for peer re- 
viewers to perform their duties? Workshop participants answered this ques- 
tion in a number of ways. In their presentation of the main findings from 
an evaluation of the peer review system at the former Office of Educational 
Research and Improvement (OERI) during the mid-1990s, Diane August, 
senior research scientist, Center for Applied Linguistics, and Penelope 
Peterson reported on an analysis of the fit between the expertise of review- 
ers and the competitions they reviewed for. Using the standards for peer 
reviewers that were in place at the time, they focused on the extent to 
which each reviewer had content, theory, and methodological expertise. 
They found a number of disconnects, including a relatively low level of fit 
on the methodological aspects of the research proposals under review (Au- 
gust and Muraskin, 1998). 

Expertise is required in three main areas to identify high-quality edu- 
cation research in the review process: the content areas of the proposed 
work, the methods and analytic techniques proposed to address the re- 
search questions, and the practice and policy contexts in which the work is 
situated. 

At one level, it is self-evident that reviewers need to know something 
about content to review education research proposals. But “education” is a 
term covering a vast territory of potential areas of study. Some competi- 
tions for research dollars are cast quite broadly (e.g., early childhood devel- 
opment), while others carve out a well-defined subtopic (e.g., effectiveness 
of pre-K curriculum on school readiness). Content expertise, then, is de- 
fined by the research priorities in the competition itself. Even in relatively 
circumscribed competitions, a wide range of content knowledge is typically 
required to adequately judge the merits of a set of proposals. Furthermore, 
the knowledge of content as it applies to teaching and learning that content 
is important. Referencing Shulman (1986), Borko made this point at the 
workshop, asserting that in order “to review proposals about mathematics 
teaching and learning, [reviewers] really do need to know about mathemat- 
ics, and . . . teaching and learning. Pedagogical content knowledge is kind 
of the nexus of those aspects of knowledge.” 

Another dimension of expertise necessary for peer review of education 
research proposals is knowledge of relevant methodological and analytic 
techniques. Like any profession, familiarity and facility with the tools of 
the trade are an essential part of the job. Reviewers must posses a solid 
grounding in methodological approaches best suited for studying the par- 
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ticular problems or topics reflected in the competition. Competent peer 
review of the quality of research must be conducted by groups of research- 
ers who are together familiar with both general standards (like those out- 
lined in Scientific Research in Education, National Research Council, 2002) 
and specific standards (relative to particular subfields) and who practice 
these standards in their own research studies (National Research Council, 
1992; Chubin and Hackett, 1990; Cole, 1979). 

Finally, reviewers must be grounded in the overarching practice and 
policy contexts associated with the area under consideration. This founda- 
tion is necessary to place the potential contribution of new work in the 
context of current issues and problems facing education policy makers and 
practitioners, as well as to consider the kinds of expertise that might be 
required to carry out the work effectively. 

Do all reviewers need to have each kind of expertise to participate 
effectively? Most workshop participants agreed that not only was it nearly 
impossible to find people with such breadth and depth of experience and 
expertise, but also that it wasn’t necessary. Rather, we agree with most par- 
ticipants that it is the combined expertise of the group that matters. That is, 
constructing panels with appropriate expertise requires ensuring that the 
group as a whole reflects appropriate coverage. Hackett made this point 
most directly, arguing that it is the “distributed” expertise on a peer review 
panel that is relevant. 

Beyond these three broad areas of competence that we view as essential 
for peer review panels, additional kinds of expertise relevant to the process 
surfaced in workshop discussions. For example, Robert Sternberg, director 
of the Yale Center for the Psychology of Abilities, Competencies and Ex- 
pertise and the president of the American Psychological Association, sug- 
gested that creativity is an undervalued yet critical talent for assessing re- 
search quality. 2 

Teresa Levitin, director, Office of Extramural Affairs, speaking from 
her experience running panels at the National Institute on Drug Abuse at 
the National Institutes of Health (NIH), referred to a number of personal 
qualities that make for effective reviewers. Such people listen respectfully 
and are intellectually open to genres of research outside their realm of 
expertise. They neither dominate nor acquiesce during face-to-face delib- 



2 Due to illness, Sternberg did not attend the workshop but sent his presentation slides 
for the committee’s consideration. 
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erations about proposals under review. Although we deem these traits as 
secondary to the three dimensions of expertise we describe here, they are 
some of the intangibles that influence the success of the peer review pro- 
cess in a very real way and therefore must be considered in vetting reviewer 
candidates. 



Conflicts of Interest and Bias 

For peer review to be an effective tool for identifying and supporting 
high-quality research, it must be credible. Essential to the integrity and 
legitimacy of the process is ensuring that reviewers do not have a vested 
interest in the outcomes of the competition that could introduce criteria 
other than quality into the process. Thus, it is essential to vet potential 
reviewers for whether they would have a conflict of interest that would 
prevent them from fairly judging a proposal or set of proposals. At one 
level, it is the responsibility of agency staff to probe these potential prob- 
lems. But it is also a critical part of an ethical code of conduct among 
investigators to be forthcoming about their relationships to the proposed 
work. As Levitin put it: “the integrity of the system really depends on the 
integrity of the individual reviewers.” 

Conflicts of interest may arise in situations in which there is a possibil- 
ity, or a perceived possibility, that a reviewer, or his or her associates, might 
gain from a decision about funding. Agencies deal with these issues in dif- 
ferent ways. Steve Breckler, of the Social Behavioral and Economic Sciences 
directorate at the NSF, referenced a “complex array of conflict of interest 
rules” that applies to peer review of research proposals submitted to the 
NSF. Brent Stanfield, deputy director, NIH’s Center for Scientific Review, 
mentioned that applicants for funding from the NIH are encouraged to 
identify “competitors” who they feel would be too influenced by the out- 
come of the review to serve as fair reviewers, and that panelists with poten- 
tial conflicts of interest on a particular proposal would recuse themselves 
from the discussion of its merits. Louis Danielson, director of the Research 
to Practice Division, Office of Special Education Programs (OSEP), de- 
scribed the interpretation of these and related rules by the U.S. Depart- 
ment of Education that preclude the participation of reviewers with par- 
ticular affiliations. 

A related but distinct idea that shapes the vetting of panelists is bias. 
Biases are preferences that may influence the degree to which proposals are 
judged fairly. Everyone has preferences, and researchers are no exception: 
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their own work and participation in a field frames the way they view the 
world. The danger comes when these preferences preclude a careful and 
open-minded reading of approaches that diverge from a reviewers personal 
viewpoint. 

As important as it may seem to identify and eliminate conflicts of 
interest and biases in the peer review process, enhancing the likelihood that 
the system identifies and supports high-quality research renders the pursuit 
of these absolute goals unattainable and unadvisable. In making decisions 
about who to include on panels, many top-flight investigators predictably 
have potential conflicts or biases: they are likely to be very familiar with 
each other, and they may have collaborated on projects, critiqued each 
other’s work, coauthored papers, or mentored or taught an applicant. At a 
minimum, they are likely to have already formed views on each other’s 
work. These biases reflect the preferences that investigators have for certain 
theoretical and methodological practices and their ideas of what the cutting 
edge in a field is or should be and therefore affect the ways in which pro- 
posals are viewed from the outset. 

The existence of these relationships and viewpoints raises questions 
about the impartiality of reviewers to judge the merits of a proposal fairly 
that must be addressed in vetting investigators for participation on panels. 
However, if a decision rule regarding conflicts of interest is applied too 
stringently, the pool of competent reviewers will dwindle significantly. Mak- 
ing conflicts of interest public is essential, but eliminating them altogether 
is not feasible. And while conflicts of interest should be minimized, it is 
often the case that agency personnel need the flexibility to exercise their 
judgment about how to carefully balance the imperative of involving top 
experts in the process while guarding against reviews that are based on 
judgments outside the merits of the proposals themselves. 

With respect to bias, however, the issue for assembling panels is to 
achieve a balance of perspectives and biases. The goal is not to minimize 
biases — as they are inherent in every reviewer — but rather to ensure that no 
single paradigm or perspective dominates the review panel. As we argue in 
the section that follows, engaging a range of perspectives sharpens thinking 
about, and opens avenues for considering, quality in the research that is 
funded. And as we discuss in the section on quality, so long as reviewers can 
agree on basic standards of quality, these divergent preferences can be ac- 
commodated in the peer review process and indeed can strengthen its out- 
comes. Without this common framework, however, there is no basis for 
negotiating differences in productive ways. 
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Diversity 

Two broad types of diversity are relevant to assembling high-quality 
panels and to promoting education research quality through peer review: 
diversity of disciplinary and methodological perspectives and diversity of 
groups traditionally underrepresented in education research. Actively pur- 
suing diversity along both of these dimensions in an agency’s peer review 
system can serve a number of important functions, including lending the 
process legitimacy, enhancing and extending learning opportunities in peer 
review deliberations, and promoting the identification and support of high- 
quality research. We take up the first two of these functions in later sections 
of the report, focusing on the discussion of quality in this section. 

Engaging peers with a range of scholarly perspectives is important for 
assessing quality in any field, including education research. Redish, draw- 
ing on his experience in physics research as well as in physics education 
research, cautioned on the dangers of peer review systems having a narrow- 
ing effect on a field too quickly. He argued that peer review systems ought 
to reflect an ethos of scientific “pluralism,” especially in a field like educa- 
tion research that is multidisciplinary and still emerging as an area of scien- 
tific inquiry. 

Assembling diverse panels with respect to groups traditionally 
underrepresented in education research — like racial and ethnic minorities — 
is also an important consideration that surfaced a number of times in work- 
shop discussions and is especially relevant to education research, as it often 
grapples explicitly with issues involving diversity. One important aspect of 
research quality across many of the agencies discussed at the workshop is 
the relevance or significance to educational problems of the proposed work. 
Assembling panels with a range of personal backgrounds and experiences 
can foster an environment in which questions are provoked and issues raised 
that otherwise might not have surfaced, and help ground the review in the 
cultural and social contexts in which the work is proposed to be conducted 
and expected to have an impact. 

Vinetta Jones, dean of the Howard University School of Education, 
made this point directly in posing questions to Grover (Russ) Whitehurst 
about the diversity of peer review panels at the Institute of Education Sci- 
ences (IES). She argued that pursuing excellence in, and specifically ensur- 
ing the relevance of, education research projects and programs requires an 
inclusive approach to the composition of panel membership with respect to 
racial and ethnic diversity, gender, and other background characteristics. 
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Whitehurst responded by relaying his personal experience reviewing the 
publication record of potential peer reviewers, noting that their racial and 
ethnic background was rarely evident. He agreed that it was essential to 
ensure that deep knowledge of the populations and contexts in which edu- 
cation research would be conducted is represented in peer review delibera- 
tions, but that he seeks to ensure that peer reviewers have this knowledge 
by reviewing the focus of their previous publications. 

These differing viewpoints and strategies underscore the complexities 
associated with the relationship between quality and group membership on 
peer review panels. While expertise and the personal background character- 
istics and experiences of panelists are different constructs, they are often 
related, at least at this point in history. For this reason, in the long run, we 
think it is likely that socially and culturally diverse peer review panels will 
result in a more expansive set of perspectives on the assessment of relevance 
and significance, thereby improving the overall quality of the research over 
time. Since quality in the peer review of education research proposals in- 
cludes both technical and relevance criteria, ensuring a diverse set of panel- 
ists who collectively bring the expertise and experience necessary to judge 
both well should always be the goal. 



Practitioners and Community Members as Peers 

Should practitioners — for example, state school officers, superinten- 
dents, principals, teachers, curriculum developers — be peer reviewers? This 
is a hotly contested question in many domains of research, one that also 
pertains to the diversity in perspectives in peer review panels. Various coun- 
tries and institutions have approached this question in different ways. For 
example, the Dutch Technology Foundation includes “lay citizens” in their 
reviews (Hackett and Chubin, 2003). Other institutions have devised in- 
novative ways to involve community members in their work outside the 
peer review process itself. For example, Harold Varmus, former NIH direc- 
tor, tried to bridge gaps between researchers and community members by 
setting up a Director’s Council of Public Representatives. The council brings 
together representatives from various groups with an interest in medical 
research, such as patients and their families, health care professionals, and 
patient advocacy groups, to advise and make recommendations to NIH on 
issues and concerns that are important to the broad development of NIH 
programmatic and research priorities. If one aspect of the quality of educa- 
tion research — as we have argued — is its connection and relevance to policy 
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and practice, then it would stand to reason that those closest to the practice 
of education ought to bring their expertise directly to the task. 

Following this logic, the former OERI and the EHR Directorate at 
NSF have historically tapped the expertise of practitioners and other stake- 
holders (e.g., parents) by including them as peers alongside researchers in 
reviewing the merits of education research proposals. Many workshop par- 
ticipants, however, questioned the implementation of this strategy, and the 
experience of several committee members led them to raise concerns dur- 
ing the event as well. For example, in their evaluation of peer review panels 
in the former OERI described at the workshop, August and Muraskin 
(1998) found that while most of the reviewers in their sample had con- 
ducted research in education, a sizable minority had not. In his remarks. 
Dodge warned that asking individuals without research expertise to evalu- 
ate scientific quality “discredits the process.” And Hackett, while arguing 
that peer review in education is a natural place to help bridge policy and 
practice, acknowledged that practitioner (those without research expertise) 
participation on review panels could undermine attempts to develop a 
strong sense of professional culture in the field. It may also serve to intro- 
duce political criteria into the review of merit if, for example, advocates 
participate on panels. 

As Hackett suggested, however, there are also benefits associated with 
engaging the viewpoints of practitioners and stakeholders in peer review 
panels. Practitioners and stakeholders are typically well qualified to discuss 
the relevance of a particular proposal and its potential contributions to 
practice. They may also have comments about the application of an inter- 
vention proposed for evaluation. Although they are less likely to have ex- 
pertise on specific technical aspects of the proposal, such as the design, 
statistics, and sampling plan, they may provide insights about relevant fea- 
sibility concerns. 

Moreover, in one of the few studies of the impact of research consum- 
ers and advocates on peer review panels, Andejeski et al. (2002) reported 
that both researchers and consumers found it highly valuable to include 
consumers (in this case, survivors of breast cancer) on peer review panels 
for the Department of Defense research program on breast cancer. How- 
ever, in contrast to the way in which practitioners and stakeholders have 
often been incorporated into education research panels, the ratio of scien- 
tists to lay reviewers was high (averaging about 7:1), consumers were trained 
on the criteria and the process, and they were assigned specifically to review 
the applications for the importance and applicability of the research and 
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issues related to human volunteers, such as the burden on the participant. 
The consumers made their comments after the scientists’ review. 

Overall, workshop participants and the committee agreed that the par- 
ticipation of practitioners in the education research and review process was 
critical; whether and how agencies involved practitioners in peer review 
panels to accomplish that goal varied considerably across and within agen- 
cies. For example, NIH has a two-tiered model. First, study sections (most 
often convened by the Center for Scientific Review), consisting of scientific 
expert reviewers, judge the scientific merit of proposals. The result of the 
review is a score and a written summary of the evaluation. Second, insti- 
tute-specific advisory councils, composed of both scientists and other stake- 
holders, consider the relevance of the proposals, and in view of both the 
scientific merit and the potential impact, make recommendations about 
which proposals should receive funding. 

Still another way to systematically engage practitioners in reviewing 
research is through an approach used by OSEP, whose agency assembles 
peer review panels of stakeholders to retrospectively assess the value of the 
agency’s portfolio of research in addressing practical ends. This structure, 
when coupled with peer review by researchers, captures the expertise of 
both but does not involve practitioners in judging the merits of research 
proposals directly. 

Finally, several agencies include practitioners on priority-setting over- 
sight boards. While separate from the peer review process itself, the identi- 
fication of areas ripe for research shapes the content of the research compe- 
titions and the proposals received in response, indirectly but significantly 
influencing the policy and practical grounding of the research. For ex- 
ample, the former National Educational Research Policy and Priorities 
Board and the new National Board for Education Sciences are both mod- 
eled on this idea. 



How Quality Is Judged 

Evaluation criteria — how potential research quality is operationalized 
for the purpose of peer review — focus the review on specific dimensions of 
quality. The criteria used to judge research proposals vary across agencies 
and sometimes across competitions within agencies. All include some as- 
sessment of technical quality or scientific excellence (“intellectual merit,” 
“quality of design,” “approach”) and typically its relevance (“significance,” 
“broader impacts”). Agencies commonly weigh and quantify these criteria 
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to ensure that no proposal would get a high total rating if it scored low on 
either. As Breckler put it at the workshop, technical merit is necessary but 
not sufficient; similarly, relevance is necessary but not sufficient. Some agen- 
cies also consider the quality of the personnel and management plan (e.g., 
for larger projects like research centers that include multiple investigators 
and institutions). Other systems include an overall judgment of quality as 
well. For example, Danielson, in describing the peer review process at OSEP, 
said that reviewers score proposals on a 100-point scale, but they are also 
asked to give an additional recommendation of “approved, disapproved, or 
conditionally approved.” Similarly, the NIH study section assessments in- 
clude “approval” or “disapproval” as well as overall judgments of quality 
(e.g., “outstanding,” “excellent,” etc.). 

In most agencies, peer reviewers are asked to assess each proposal 
against these criteria, to assign corresponding scores as appropriate, and to 
provide written comments to support their scores and describe strengths 
and weaknesses in each proposal. Peers discuss their views and scores as a 
group, and the opportunity to change scores based on group discussion is 
extended. Once final scores are assigned, staff averages the scores, creates a 
slate of proposals ranked from highest average score to lowest, and forwards 
the slate to the head of the agency for final sign-off and funding decisions. 

Ensuring quality along the dimensions used by an agency suggests the 
need to create measures that are both reliable and valid. Reliability in this 
context refers to the extent to which a research proposal would receive the 
same ratings, funding outcome, and feedback across multiple independent 
review panels. Ensuring high reliability is important because it helps to 
quell fears that the ratings are an anomaly or just a function of the particu- 
lar group assessing them. Even if ratings are perfectly reliable, however, they 
may not reflect the intended evaluation criteria — that is, they may not be 
valid. Reliability does not ensure validity, but without reliability, results and 
feedback will be inconsistent and almost surely not valid. 

At the workshop, Domenic Cicchetti, statistician and methodologist 
at Yale University and author of seminal publications on the topic of reli- 
ability in peer review, provided an overview of his work on reliability in the 
evaluation of both journal submissions and grant proposals, based on an 
annotated presentation he prepared for the workshop (Cicchetti, 2003). 
Analyzing agreement statistics across individual judges involved in peer re- 
view of manuscripts submitted to journals for publication, he concluded 
that reliability was generally low. 
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How to think about and promote reliability in any form of peer review 
is a topic of considerable controversy and commentary (see, e.g., extensive 
commentary on Cicchetti’s foundational work in this area in an issue of 
Behavioral and Brain Sciences, 1991). In our view, examining agreement 
among individual judges is reasonably appropriate for assessing reliability 
in journal submissions, because reviews are typically conducted indepen- 
dently by mail and then simply averaged. For our purposes in considering 
research proposal reviews, however, the review process more typically in- 
volves group discussion among panelists with different types of expertise 
and is designed to promote consensus. Since the process necessarily in- 
volves interaction and argument, the individual ratings among panelists are 
not independent. In fact, a panel with diverse content and methodological 
expertise will be likely to produce a more complete review even though 
initial ratings by individual panel members may vary widely (that is, be 
inconsistent with one another). To the extent that the consensus-building 
processes are effective, analyses of initial independent ratings may underes- 
timate the reliability of group results — that is, they may be poor indicators 
of the reliability of the group consensus on quality as reflected by group 
expertise. However, the reliability of panels as a whole, while a more useful 
construct, is difficult to measure because agencies overseeing reviews of 
research proposals never have the luxury of convening multiple panels to 
review the same proposals and then comparing the results across the inde- 
pendent panels. 

Validity, as applied to the results of peer review, refers to the extent to 
which inferences made from the resulting ratings and specific feedback are 
warranted given the information provided in proposals for research fund- 
ing (Messick, 1989). It is possible for results to be reliable (consistently 
repeatable) but still not support valid judgments of the merits and deficien- 
cies of a proposal. In general, validity is considerably more difficult to assess 
than reliability, and there have been very few studies of the validity of peer 
review results. 

Evidence for validity will vary across the different priorities and evalu- 
ation criteria established by different agencies. NSF programs, for example, 
use two separate criteria: intellectual merit and broader impacts. Measures 
of validity for intellectual merit ratings might include the extent to which 
ratings reflect how well relevant theoretical constructs are characterized in a 
proposal, or the appropriateness of applying a particular statistical test for 
analyzing the data that will be collected. Assessing the validity of impact 
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ratings might involve examining whether they predict the actual participa- 
tion of traditionally underrepresented groups in funded projects to a useful 
extent. Few agencies have the time or resources to invest in true validity 
studies. The difficulty of establishing the validity of peer review results em- 
pirically is, in fact, the major reason why the use of expert judgment is the 
single best option for proposal evaluation. 

Finally, there is an element of quality considerations in peer review 
that relates to risk. Some agencies incorporate the idea of originality or 
innovation into the criteria used to assess quality. Indeed, in a recent study, 
Guetzkow, Lamont, and Mallard (2004) found that multidisciplinary so- 
cial science peer review panelists often viewed originality as what distin- 
guished worthy from less worthy academic work. Although this idea was 
not explored in much depth at the workshop, it is an important consider- 
ation. Risk can be thought of as a dimension of quality with respect to the 
broad education research portfolio in an agency. If agencies never support 
new work that strikes off in a new direction, develops new methods or 
analytic tools, challenges core ideas, or approaches a problem from a novel 
perspective, the potential for significant progress, or even breakthroughs, 
will be substantially curtailed. However, peer review tends to reward pro- 
posals that rely on established assumptions, models, and techniques. Risk- 
taking, therefore, may have to be supported through other funding mecha- 
nisms, but so long as it is undertaken to strategically invest in highly 
innovative work, it can be an important element of federal education re- 
search portfolios. 

Workshop discussions about research quality analyzed both short-term 
and long-term aspects of quality, and many participants argued that peer 
review systems ought to be designed to attend to both. Peer review is typi- 
cally designed to identify high-quality proposals for a given agency compe- 
tition. But quality can also be viewed as a long-term prospect. Both Redish 
and Borko explicitly isolated the potential for peer review to upgrade the 
future quality of research. Indeed, it could well be that none of the propos- 
als submitted in a particular competition will lead to research of the 
highest quality. In this case, the only way to improve the quality of educa- 
tion research is to get authors to improve the quality of their proposals. 
Even when research is funded, feedback on issues requiring additional at- 
tention can provide constructive suggestions on how to upgrade future 
submissions. 



Copyright © National Academy of Sciences. All rights reserved. 



Strengthening Peer Review in Federal Agencies That Support Education Research 
http://www.nap.edU/catalog/1 1 042.html 



ANALYZING KEY ELEMENTS 39 

FURTHER DEVELOPING A 
PROFESSIONAL CULTURE OF INQUIRY 

Peer review of education research proposals also ought to be designed 
to support the development of the field of education research. In this sec- 
tion, we analyze facets of peer review that relate most directly to upholding 
this objective: diversity of perspectives and backgrounds, standing panels, 
feedback, the role of staff, and training. 



Diversity 

Several workshop participants suggested that since peer review can 
and should serve an educative function, efforts to involve a diversity of 
research perspectives as well as the participation of people from tradition- 
ally underrepresented populations in the process were imperative. In re- 
sponse to a question about how agencies ensure diverse perspectives on 
peer review panels, Steven Breckler told the group that NSF program of- 
ficers spend a significant amount of time trying to identify people and 
places that “ordinarily are not plugged into the NSF review process.” He 
also pointed to the NSF criteria for reviewing research applications, which 
require an assessment of the extent to which the proposed activity will 
broaden the participation of such groups in the evaluation of the proposals 
themselves “broader impacts.” According to Stanfield, NIH also pays close 
attention to these issues, relying on a number of mechanisms to promote 
broader participation, including the use of discretionary funding to sup- 
port research among underrepresented groups and institutions. 

In terms of this professional development goal, workshop discussions 
also focused on the role of peer review for developing junior scholars, an- 
other way to view diversity in the composition of panels. At the workshop, 
Peterson argued that a critical function of peer review in education research 
was to promote learning opportunities and growth among early career re- 
searchers. Borko made a similar argument, suggesting that peer review be 
used to “mentor the next generation of researchers.” Agency representatives 
offered examples of how this goal is pursued in practice. For example, 
Sloane noted that in his work, “we make an effort to have about 20 to 25 
percent of our panels be people who are not tenured.” 
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Standing Panels 

Panelists can be assembled once to review a single set of proposals (ad 
hoc panel) or on a regular basis to meet over a predetermined length of 
time and consider a particular area of research (standing panel). There are 
strengths and weaknesses of both approaches. Ad hoc panels may be pru- 
dent when efficiency must be maximized; the review of small, exploratory 
grants may also be best served by assembling one-time groups. 

To promote professional development and capacity building in the 
field, standing panels are a very attractive mechanism. Since education re- 
searchers come from so many fields and orientations, panels focused on 
particular issues or problems in education can promote a collective exper- 
tise that builds interdisciplinary bridges and facilitates the integration of 
knowledge across domains. Hackett, drawing from his own experience par- 
ticipating on NSF peer review panels, asserted that establishing interdisci- 
plinary trust is difficult when panels are ad hoc. In contrast, he argued that 
standing panels that convene groups of investigators regularly around is- 
sues or problems can be quite promising in this regard. Standing panels 
provide a context for researchers to build relationships with scholars they 
might not otherwise know. Panel members can carry these experiences into 
their own work and that of their colleagues, forging broader disciplinary 
connections among more and more researchers studying common phenom- 
ena and questions but approaching them from different perspectives. 

The use of standing panels is also likely to encourage the participation 
of top-flight investigators, as these longer term experiences are more attrac- 
tive as professional learning opportunities than short-term panels. Offering 
this benefit is particularly needed in education. In their evaluation of OERI, 
Diane August and Lana Muraskin reported that many former panelists do 
not view peer review as worthwhile for their career development and trajec- 
tory (August and Muraskin, 1998). Although there are surely many factors 
that lead to this sentiment, it is worth noting that peer review panels at 
OERI were always ad hoc. 

Standing panels can also provide the kind of stability and institutional 
knowledge that can facilitate positive outcomes in resubmitted proposals. 
Not all agencies have standard resubmission policies — that is, formal pro- 
cedures that unsuccessful proposers can follow to respond to the reviews of 
the proposal and potentially receive funding at a future date. Such pro- 
cesses can identify promising projects in need of further development for 
funding and provide concrete direction for improvements in specific areas. 
When an (improved) application is resubmitted, the panel members know 
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the history of its development and can more knowledgably evaluate it on 
how well the proposers have responded to specific critiques rendered dur- 
ing its initial review. 

In addition, when groups of scholars meet regularly in peer review, 
they provide continuity of vision to programs of research — lines of inquiry 
in particular areas that together point to new insights, raise new questions, 
and suggest future directions for agency competitions. Over time, panelists 
acquire an understanding of the roles and relationships between the field 
and the agency, enhancing mutual understanding and reinforcing the norms 
of the culture in the context of the agency’s operations. It is the continuity 
that standing panels bring to an agency’s peer review system that is the basis 
for fostering powerful learning among proposers, reviewers, and staff. 

Although well-suited as a professional development tool, standing pan- 
els have their drawbacks. Retaining the same people over time can have a 
narrowing effect on the advice given to agency leadership, which is why 
many standing panels have term limits. Standing groups develop a consen- 
sus view of the field and its needs, which can result in neglecting potentially 
important lines of inquiry, methodological approaches, or contextual fac- 
tors. Worse, they can institutionalize the biases the members bring to the 
work. The potential for these negative consequences is heightened if the 
members are not explicitly and carefully selected to represent a range of 
perspectives, if they do not approach their work with a willingness to listen 
and to consider differences of opinion and approach thoughtfully, and if 
their biases are not declared, considered, and balanced. 



Feedback 

Most peer review systems are designed in one way or another to pro- 
vide substantive feedback to proposers (or would-be proposers) on the 
strengths and weaknesses of their plans. The mode of feedback can take any 
number of forms. At the Office of Naval Research (ONR), for example, 
program officers spend substantial amounts of time working directly with 
investigators before they write a formal proposal for funding consideration. 
At many other agencies (e.g., NIH, NSF, and OSEP), the primary feedback 
mechanism is the provision of written products from the proposal review 
process — forms completed by reviewers that detail strengths and weaknesses 
for each evaluation criteria. 

Substantive feedback — as well as clear guidelines for resubmission of 
rejected proposals — can play a vital role in promoting peer review’s educa- 
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tive function. At the workshop we learned that a major finding of the OERI 
evaluation by August and Muraskin (1998) was that the written reviews of 
proposals were cursory and often merely descriptive summaries of the con- 
tent of the proposals themselves (rather than analysis of the content with 
respect to the review criteria). Both Borko and Peterson emphasized the 
value of feedback in the process and the need to upgrade its use in current 
systems. At the same time, agency staff from OSEP cited persistent prob- 
lems getting reviewers to fully document their comments and to clearly 
justify their ratings, and August and Muraskin (1998) noted this problem 
in their evaluation of the former OERI’s peer review system as well. If peer 
review is to serve a professional development function effectively, agency 
staff and reviewers should take these responsibilities seriously and invest the 
time to fulfill them. 

Yet another issue aired at the workshop showed how difficult establish- 
ing high-quality feedback can be. Both representatives from NIH described 
difficulties the agency encountered in recent years because investigators 
bristled at what they perceived to be inappropriate directives from review- 
ers. In response, then-director Harold Varmus determined that summary 
statements emanating from reviews should evaluate the proposed research 
according to established review criteria, but they should not be tutorials 
telling investigators how to do their research. In this context, there was 
considerable discussion about the appropriate level of detail that ought to 
be part of reviews: How do reviewers and staff balance the need to justify 
ratings and to communicate effectively with applicants while respecting the 
professional judgment of applicants? Danielson also raised the issue of re- 
source constraints in this context, suggesting that if the agency were to 
provide detailed feedback on each of the roughly 4,500 applications they 
receive each year, they would have to contract the work out due to limited 
staff resources. We support erring on the side of more detailed information 
and critique, as this documentation is a key component of a feedback loop 
that can lead to future improvements in a field. 

In addition to reviewers’ written feedback, agency staff can also inter- 
act with members of the research community — at professional association 
meetings, workshops convened specifically for principal investigators and 
future principal investigators, and other such venues — to orient investiga- 
tors to the agency’s priorities and processes. The level of detail, approach, 
and other such particulars associated with the content and format of pro- 
posals is not the same across or even within federal research agencies, and 
the more familiar proposers and reviewers are with these important process 
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mechanisms, the better the review and, most importantly, the better the 
products of the review. Explicit training on the nature of feedback should 
also be provided to reviewers; we take up such training issues in a later 
section. 



Role of Staff 

Another key feature of peer review systems is the role of staff in the 
process. Agency staff are part of the human resources of the research field, 
playing both teaching and learning roles. There are very real trade-offs asso- 
ciated with the various models of staff involvement in practice today. Three 
of the agencies represented at the workshop — NIH, NSF, and ONR — 
nicely illustrate two models at opposite extremes and a hybrid approach to 
staff involvement. At NIH, the system is very deliberately built to erect a 
clear separation (sometimes called a firewall) between the staff who write 
the grant announcements soliciting proposals and developing scientific pro- 
grams and the staff who select and interact with peers in the review of 
proposals received in response to those solicitations. In contrast, at ONR, a 
single staff person (sometimes called a strong manager) performs all of these 
functions. The system at NSF falls somewhere in between — endowing pro- 
gram officers with a fair degree of authority to shape competitions and to 
select peers, while creating checks and balances in the system to guard 
against improprieties. 

The benefit of the ONR approach is in continuity of expertise. Knowl- 
edgeable staff can follow the process from beginning to end, substantively 
interacting with members of the field in ways that facilitate learning on 
both sides and result in work with tight alignment to agency goals. As 
Susan Chipman, director, Cognitive Science Program, of the ONR, de- 
scribed the process, “ONR staff are the peers — they review proposals and 
make recommendations for funding.” Program officers at ONR often use 
multiple internal peers to judge research proposals, including potential con- 
sumers of the work, since ONRs work is very applied and mission-ori- 
ented. Program managers like Chipman actively develop research programs 
based on the needs of their agency. 

The trade-off is that this kind of participation across all parts of the 
peer review process can result in a loss of external legitimacy. Whitehurst, 
in describing his plans for peer review at the IES, articulated this downside. 
In the former OERI, program officers who developed solicitations also se- 
lected the peers to review proposals. He acknowledged that this continuity 
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is beneficial because that person becomes expert in all aspects of the com- 
petition. The problem, as he described it, is that having responsibility for 
both kinds of tasks raises the possibility of infusing bias into the system, 
thereby weakening its overall legitimacy. As he put it, investigators might 
reasonably wonder: Is everyone getting a fair shake, or are those researchers 
who are chummy with the program officer getting an unfair advantage? 
The NIH model, with its built-in firewall, creates a clear boundary and, as 
Dodge put it, this “keeps it pure.” 

Describing the NSF process as it relates to these two models, Breckler 
argued that their hybrid approach taps the best of both worlds by relying 
on external panels of experts while allowing program officers substantive 
involvement. He asserted that the tenets of social psychology suggest that 
the best way to get people to act responsibly is to make them identifiable 
and responsible for what they are doing, supporting the kinds of roles that 
staff are authorized to serve: crafting program announcements, selecting 
peers for review, and settling on a slate to pass on for funding decisions. 
This approach, he suggested, allows one person to go against the group 
tendency to be conservative — that is, to reject innovative ideas. And a high 
level of responsibility helps to attract high-quality officers to the agency. 

Responding to questions about the potential abuses of such a system, 
Breckler argued that the system is rarely compromised because the process 
is open. The agency mandates extensive documentation of peer review pan- 
els, requiring program officers to certify that they have completed parts of 
the process to the best of their ability and in concordance with relevant 
policy. To address charges that some investigators may not get the fair shake 
to which Whitehurst referred, Breckler pointed to a complex array of con- 
flict of interest rules for program officers. Furthermore, NSF has a long- 
standing tradition of instituting a final check in the process by engaging a 
committee of visitors to periodically and comprehensively assess research 
programs on a host of dimensions, including whether such conflict of in- 
terest rules were followed. The researchers who are called on to serve this 
function are asked to carefully scrutinize all aspects of the process to assess 
its fairness and legitimacy, and the results of the assessment are made pub- 
licly available. 



Training 

For peer review to fulfill a professional development function, explicit 
training for reviewers, proposers, and staff must be part of the process. But 
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workshop participants revealed that in-depth training is the rare exception 
rather than the rule in practice. 

Training reviewers was raised repeatedly at the workshop as an impor- 
tant element of the peer review process, and agency participants discussed 
strategies and identified impediments to facilitating successful training. 
Stanfield described ways NIH helps to familiarize reviewers with the peer 
review process, including brokering meetings prior to the panel discussion 
and setting its tone by beginning with experienced reviewers. Breckler as- 
serted that providing model reviews to reviewers would be a helpful strat- 
egy, lamenting that this practice is not permitted at NSF. Chipman agreed, 
suggesting that the use of model reviews could help strengthen a tradition 
of high-quality reviews in peer review settings for education research. 

In describing some training techniques she has used for reviewers at 
the National Institute on Drug Abuse at NIH, Levitin highlighted several 
potentially helpful strategies. She suggested that training starts well before 
the first meeting of the group, is both formal and informal, and is grounded 
in “general principles and policies.” Levitin suggested that if reviewers are 
well versed in a “few fundamental” ideas, they will be able to provide a fair 
review. She made clear that there cannot be hard and fast rules for every 
circumstance, given the very complex nature of review, different types of 
applications, and other factors, but that there are policies and procedures to 
guide review in making fair judgments. One key area of training she de- 
scribed relates to teaching reviewers how to apply the review criteria. At 
NIH, ratings range from 1 to 5, with 1 being the most meritorious. Levitin 
also stressed that it is important to communicate to reviewers how to pro- 
vide balanced and thorough reviews, so that the strengths and weaknesses 
of every application are described and only the stated review criteria are 
used to assess them. 

Training potential applicants was also an area discussed at the work- 
shop. The agencies represented at the workshop relied on a range of largely 
informal strategies to promote better proposals — such as program officers 
talking with junior scholars about the grant-writing process — and the de- 
gree to which this issue was addressed varied quite a bit. Procedures for 
resubmission at NIH was the most formal procedure described: with clear 
and comprehensive written feedback on the weaknesses of a submission, 
proposers get insights into how to improve their future proposals to the 
agency and are informed of specific guidelines for resubmitting a revised 
application in a future grant cycle. Milton Hakel, an industrial and organi- 
zational psychologist from Bowling Green State University, suggested that 
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the ability to write rejoinders to reviews could also be instructive. In many 
respects, the opportunity to revise an application in response to peer review 
provides this type of opportunity. One-time submission policies without 
explicit requirements for identifying a proposal as a resubmission and ex- 
plaining how the grant has been revised misses valuable opportunities for 
professional development of the researchers. 

Finally, the training of staff is similarly important, but no one at the 
workshop mentioned any kind of professional development for staff in- 
volved in peer review systems. Indeed, in their recommendations, August 
and Muraskin (1998) suggested staff training as a strategy for improving 
the peer review process at OERI. How to develop training for agency staff 
would depend on the specific tasks the staff are expected to perform and 
the skills and knowledge needed to accomplish them effectively. 

AGENCY MANAGEMENT AND INFRASTRUCTURE 

Like any system, the peer review process must be effectively managed. 
Negative experiences of many reviewers of education research proposals — 
especially in the competitions studied in the evaluation of the former OERI 
by August and Muraskin (1998) and in testimony about peer review at 
OSEP to the President’s Commission on Excellence in Special Education 
(2002) — in large part derived from poor logistics. Active, careful attention 
to logistical arrangements enables a smooth peer review process that en- 
courages participation and improves its outcomes. For example, lead time 
is critical to engaging top scholars in the process. Last-minute planning 
(often deriving from either legislative or executive branch delays) invariably 
leads to conflicts with previous commitments, seriously reducing the likeli- 
hood of tapping top talent to participate. It also leaves little time for sub- 
stantive reflection on proposals, leading to cursory and incomplete feed- 
back and, in extreme cases, poor advice to decision makers about funding 
priorities. Infrequent and inconsistent announcements can set off a “now 
or never” mentality among researchers, ensuring a high rate of rejection 
given scarce resources and depleting the pool of potential reviewers. Active 
proposal management — through triage processes that involve an initial cut 
through the proposals and assignment of only promising projects to re- 
viewers — can minimize workloads, focusing attention on high-priority ar- 
eas and making participation manageable for reviewers. 

Despite the many anecdotes of how important peer review is to the 
field and to individual research careers, agency representatives consistently 
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pointed to increasing difficulties in recruiting reviewers. Stanfield identi- 
fied the logistical hurdles involved with convening face to face meetings as 
particularly problematic: “It is very difficult to get very busy scientists to 
come to Washington three times a year for four years.” Similarly, Breckler 
stated, “it is difficult to get people who are going to dedicate themselves to 
do peer review” and “it is getting increasingly difficult.” Incentives for schol- 
ars to serve as peer reviewers derive from a number of sources and compel 
individuals to behave in a variety of ways. Many of the sources are outside 
the control of any given federal agency (e.g., whether or not service on peer 
review panels is recognized in promotion and tenure decisions). Agencies 
can do their part to enable the recruitment of top-flight investigators to 
review by ensuring that their systems are managed effectively and reviewer 
workloads are minimized to the extent possible. For example, the August 
and Muraskin (1998) evaluation reported that many reviewers at the former 
OERI spent far longer reviewing than the estimated time commitment 
they had been provided by agency staff. 

FLAWS AND ALTERNATIVES 

To this point, we have not taken on what might be considered the 
threshold question: What are the drawbacks to peer review as a mecha- 
nism for informing resource allocation of federal research dollars, and 
are there viable alternatives? There are indeed problems with peer review, 
some of them significant (see Finn, 2002; Horrobin, 2001; McCutchen, 
1997). And there are other ways that research dollars have been and are 
distributed. 

The workshop discussions did not address these questions in any de- 
tail. Hackett and Chubin’s paper (2003), however, does provide an over- 
view of some of these issues. To set the stage for the committee’s recom- 
mendations in Chapter 3, and drawing on Hackett and Chubin’s analysis, 
we acknowledge and describe some of the most worrisome weaknesses of 
peer review. We also identify some of the alternatives they describe for allo- 
cating federal education research dollars, ultimately concluding that, de- 
spite its flaws, peer review is nonetheless the best available mechanism for 
allocating scarce education research dollars. 

A persistent complaint about the peer review process is the possibility 
of cronyism — that is, that engaging peers predisposes outcomes to benefit 
friends or colleagues with no or little regard for the actual merit of a given 
proposal (Kostoff, 1994). This situation can lead to a kind of protectionism 
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that repeatedly rewards an elite few, narrowing the breadth of perspectives 
and ideas that is so critical to scientific progress and stunting potentially 
promising lines of inquiry. 

The peer review process can also inhibit innovation. Arguably, peer 
review is expected to draw the line “between sound innovation and reckless 
speculation.” As Hackett and Chubin (2003, p. 17) argue, “a review system 
at one extreme could reward novelty, risk-taking, originality, and bold ex- 
cursions in a field . . . [or] it could sustain the research trajectory established 
by the body of accepted knowledge by imposing skeptical restraint on new 
ideas.” The closer to the latter pole a system becomes, the more easily it 
could reject promising ideas as implausible. Current practice is often criti- 
cized for being too conservative — a well-known example recounted by 
Hakel at the workshop is that when the original manuscript describing the 
double-helix structure of DNA was submitted for publication, it was sub- 
jected to peer review and rejected. 

What about other ways to allocate research dollars? As Hackett and 
Chubin (2003) report, Congress has the prerogative to allocate funds 
through direct appropriation (also termed “earmarking” or “pork 
barrelling”). In fiscal year 2002, Congress earmarked $1.8 billion for 
projects at colleges and universities. While not all of this money is for re- 
search, earmarks for academia are a useful indicator of the exercise of direct 
appropriation. And while compared with the roughly $100 billion federal 
investment in research and development, $1.8 billion is a relatively small 
amount, it seems somewhat larger in comparison to the $25 billion federal 
budgets for basic research (all data from analysis by the American Associa- 
tion for the Advancement of Science of the R&D budget; http:// 
www.aaas.org/ spp/ rd / guihist.htm) . 

The main deficiency of earmarking is that it circumvents technical 
expertise, jettisoning altogether the principle that scientific quality ought 
to be the primary basis for the allocation of research dollars. It also has a 
corrosive effect on the development of the research profession: without a 
clear link between rewards (continued funding) and performance (quality 
of proposals for future work), the core values of science would be eroded 
significantly (Hackett and Chubin, 2003). 

Another alternative is to rely on a single, so-called strong manager who 
makes decisions on behalf of the agency according to his or her best judg- 
ment (as is done in ONR). As Hackett and Chubin (2003, p. 5) observe, 
“In effect, this is peer review with one peer, so this steward had better be on 
a par (intellectually and in stature within the field) with those applying for 
support . . . [and] should understand the field and its needs (which should 
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be clear and widely shared) to ensure that decisions and allocations are 
wise, legitimate, and effective.” 

The arguments offered to support the strong manager arrangement 
include that it is flexible and responsive, and an efficient way to distribute 
relatively small pots of money. It may also be appealing because the man- 
ager is held accountable for performance outcomes (e.g., research-based 
products that benefit the Navy). However, it would be nearly impossible to 
scale this approach up to the size of NIH (about $27 billion in fiscal year 
2003), and it would face similar difficulties in mid-sized research agencies. 
More importantly, such concentrated power limits the breadth and depth 
of expertise that can be brought to bear on proposals and invites serious 
questions of bias and partiality. 

Hackett and Chubin (2003) discuss a third funding alternative — using 
a formula to allocate resources. Funds may be allocated to states or univer- 
sities or institutes, then suballocated to groups or individuals according to a 
variety of additional criteria. Or formulas may be devised based on the past 
performance of individual scientists, with funds awarded accordingly. Some 
measure of current need or potential payoff may factor into the equation, 
as well as the number of researchers at a university or residents in a state. 
Fair and effective formulas would be hard to devise, and the relative merits 
of various options endlessly debated. 

None of these options for allocating research dollars is perfect, includ- 
ing peer review. When peer review is compared with these alternatives, 
however, it emerges as the mechanism best suited to promote merit-based 
decisions about research quality and to enhance the development of the 
field. This statement does not preclude some type of blended approach in 
making decisions about what research to fund, however. Indeed, maintain- 
ing a variety of funding mechanisms can be leveraged to obviate the weak- 
nesses of peer review. And there are additional design features that can be 
used in peer review to minimize potential problems. For example, the role 
of a peer review panel should always be to rank proposals, not to recom- 
mend particular decisions about what should be funded, as empowering 
panelists with making direct recommendations can more easily lead to ques- 
tions about cronyism and conflict of interest. Term limits, blended exper- 
tise on panels, and attention to systematic evaluation of peer review pro- 
cesses and outcomes are additional examples of the kind we address in 
Chapter 3 that can and should be used to counterbalance the flaws of peer 
review systems. 

In short, peer review as a system for vetting education research propos- 
als in federal agencies is worth preserving and improving. So the question 
for us is how to strengthen it — a topic we address in the next chapter. 
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Strengthening the System 



O ur guiding principle in setting forth the recommendations that 
follow derives from our dual conclusions — detailed in the previ- 
ous chapter — that peer review of education research proposals 
ought to focus explicitly on identifying and supporting the highest quality 
research and strengthening the field to foster a culture of rigorous inquiry. 

Peer review in research agencies sets a foundation for developing and 
sustaining a culture of inquiry and rigor — both within the walls of the 
agency and in the fields it supports. More so than any single process or 
practice, creating and nurturing this culture in a research agency is consis- 
tently emphasized in reports on peer review in education as well as a range 
of other fields (National Research Council, 1998, 1999b, 2002; President’s 
Commission on Excellence in Special Education, 2002). Developing this 
shared sense of commitment to best practice in an agency provides a healthy 
environment for peer review to function effectively. Specific practices in the 
peer review systems can sustain the broader culture in the field through 
reinforcing these norms among applicants, staff, and reviewers, who to- 
gether are key members of the research field. 

In our recommendations, we focus exclusively on the government side 
of the peer review partnership. We chose this perspective on the basis of an 
assessment of the current policy landscape and the near-term needs of two 
key parts of the federal government that support education research: the 
Institute of Education Sciences (IES) and the Office of Special Education 
Programs (OSEP). The president has nominated members for appoint- 
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ment to the National Board for Education Sciences of the IES, which, 
pending confirmation by the Senate, will work with the director to main- 
tain a high-quality peer review system for the agency. And the pending 
reauthorization for OSEP will almost certainly require changes in its peer 
review process. Thus, we have attempted to provide a framework for re- 
search policy makers who need to revisit their peer review systems in this 
short report. 

Overall, the recommendations we provide are of two kinds: (1) specific 
ideas to enact the conclusions we set out in the previous chapter and (2) 
our suggestions for addressing some of the problems related to such issues 
as logistics and training that were aired during our workshop. In this con- 
text, we reemphasize that the evidence we use to support these ideas is 
based primarily on workshop discussions, and our recommendations should 
be interpreted in this light. 

A final caveat: although we focus on the governments role, we should 
clarify that the responsibility for ensuring a well-functioning peer review 
system does not rest solely within the walls of federal agencies. As our de- 
piction of peer review in this report makes clear, the research communities 
as well as the broader education community all play a part in the process — 
and thus all share accountability for its success. Without the support, integ- 
rity, and participation of these communities — particularly researchers — it 
is sure to fail. 

The committee makes 10 recommendations: 

1. Peer review is the best available method to support education re- 
search funding decisions. Thus, we recommend that this mechanism be 
maintained and strengthened. 

2. Agencies that fund education research should explicitly focus on 
two key objectives for peer review: (a) to identify and support high-quality 
education research and (b) to promote the professional development of a 
culture of rigor among applicants, reviewers, and agency staff in the educa- 
tion research communities. 

3. Agencies that fund education research should develop clear state- 
ments of the intended outcomes of their peer review systems and establish 
organizational routines for ongoing, systematic evaluation of the peer re- 
view process. 

4. Agencies that fund education research should build strong infra- 
structures to support their peer review processes. This infrastructure should 
include (a) knowledgeable staff, (b) systems for managing the logistics of 
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peer review, (c) technologies to support review and discussion of proposals, 
(d) a clear mechanism for providing feedback, and (e) standing panels when 
research priorities are relatively stable. 

5. Effective peer review systems require planning and organization in 
advance of a review. In order to schedule, agencies that fund education 
research need relatively predictable levels and timing of funding. Internal 
barriers that slow down program announcements or make peer review dif- 
ficult to schedule should be minimized. To the extent possible, scheduling 
problems and complicated or burdensome logistics should be eliminated to 
support the availability and participation of highly qualified reviewers. 

6. Agencies that fund education research should uphold basic prin- 
ciples of peer review but retain flexibility in designing peer review systems 
to meet their individual needs. Agencies should be accountable for uphold- 
ing these principles and should provide data on how well their process 
achieves its goals. External mandates that extend beyond these foundations 
should be minimal, as they can hinder the development and implementa- 
tion of high-quality peer review systems. 

7. The criteria by which reviewers rate proposals should be clearly 
delineated, and the meaning of different score levels on each scale should 
be defined and illustrated. Reviewers should be trained in the use of these 
scales. 

8. As a group, peer review panels should have the research experience 
and expertise to judge the theoretical and technical merits of the proposals 
they review. In addition, peer review panels should be composed so as to 
minimize conflicts of interest and balance biases, promote the participa- 
tion of people from a range of scholarly perspectives and traditionally 
underrepresented groups, and provide opportunities for professional 
development. 

9. Agencies that fund education research should involve practitioners 
and community members in their work to ensure the relevance and signifi- 
cance of their portfolio. If practitioners and community members partici- 
pate on peer review panels, they should focus on the relevance, significance, 
applicability, and impact of an education research proposal. 

10. Agencies that fund education researchand professional associations 
should create training opportunities that educate scholars in what the peer 
review process entails and how to be an effective reviewer. 

We begin with two general recommendations that follow from our 
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overarching theme that peer review be strengthened by focusing on two 
objectives. 



KEY OBJECTIVES 

Recommendation 1: Peer review is the best available method to 
support education research funding decisions. Thus, we recom- 
mend that this mechanism be maintained and strengthened. 

A strong peer review system is a hallmark of successful research and 
development programs across the federal government. Although there are 
alternatives to the peer review process for allocating research dollars, none 
has the same potential for identifying high-quality research proposals and 
promoting the further development of a professional culture of inquiry 
among education researchers. Each of the most common alternatives — 
including a strong program manager who makes the determination, alloca- 
tions based on formulas, and legislative earmarking — inadequately ad- 
dresses both of these critical goals that peer review can serve. Peer review is 
the best mechanism to support the kinds of objectives we hold as critical 
for the future of the field and the use of education research to improve 
policy and practice in turn. 

The effectiveness of peer review depends heavily on the extent to which 
specific procedures are designed and implemented to support the stated 
objectives of the system. Thus, following the logic of the previous chapter, 
we first state the purposes that in our judgment should undergird peer 
review of education proposals in federal agencies, and then we make a series 
of suggestions for how various processes can serve them. We then recom- 
mend some basic principles for peer review systems and outline key fea- 
tures of an infrastructure that can support its functions. Peer review in- 
volves the expenditure of a good deal of resources, and it is important to 
ensure that this investment (of federal dollars and researchers’ time, prima- 
rily) is worthwhile for all. 

Recommendation 2: Agencies that fund education research should 
explicitly focus on two key objectives for peer review: (a) to iden- 
tify and support high-quality education research and (b) to pro- 
mote the professional development of a culture of rigor among 
applicants, reviewers, and agency staff in the education research 
communities. 
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Peer review practices in federal agencies have historically been designed 
based on culture and tradition. We recommend that federal agencies that 
support education research adopt two objectives — identifying and sup- 
porting high-quality education research and promoting a shared culture of 
rigor in the field — as guideposts as they design, revamp, or evaluate their 
systems. 

One main objective of peer review is to ensure that any research that is 
funded, published, or used to support education policies or practices meets 
high standards of quality (Chubin and Hackett, 1990; Hackett and 
Chubin, 2003). This goal has historically been at the center of agency peer 
review systems, and should continue to be. Upholding this goal requires 
that agencies work toward achieving clarity on what is meant by research 
quality in the context of peer review — a formidable task. Although con- 
sensus on this question has been and continues to be elusive in education 
research more generally (Lagemann, 2000), a starting point for these dis- 
cussions in peer review panels with respect to technical or intellectual merit 
criteria could be the principles of scientific education research outlined 
in Scientific Research in Education (National Research Council, 2002, 
pp. 3-5): 1 

• Pose significant questions that can be investigated empirically. 

• Link research to relevant theory. 

• Use methods that permit direct investigation of the question. 

• Provide a coherent and explicit chain of reasoning. 

• Replicate and generalize across studies. 

• Disclose research to encourage professional scrutiny and critique. 

These principles were not designed to be, nor should they be used as, 
strict standards to which individual research applications are subjected in 
peer review. 2 Central concepts embedded in the principles can and should 



! We know, for example, that background materials for reviewers of research competi- 
tions of the National Science Foundation’s Education and Human Resources Directorate 
have featured that report’s principles as guidelines for assessing quality (or, in their terminol- 
ogy, “intellectual merit”). These principles are meant to apply to scientific research; prin- 
ciples for other important modes of inquiry, such as historical and philosophical studies, are 
similarly required to ensure the quality of these types of scholarly projects and are not entirely 
dissimilar from these principles. 

2 The committee that authored Scientific Research in Education argued: “Although any 
single scientific study may not fulfill all the principles — for example, an initial study in a line 
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